2024-05-15T12:22:56.437 INFO:root:teuthology version: 0.0.1.dev271+g1ae7ad8 2024-05-15T12:22:56.437 DEBUG:teuthology.run:Teuthology command: teuthology --name rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi --owner scheduled_rzarzynski@teuthology --archive /home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707111 --verbose --description rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados tasks/rados_api_tests validater/valgrind} -- /home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707111/orig.config.yaml 2024-05-15T12:22:56.474 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-15T12:22:56.554 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707111 branch: wip-osd-ec-partial-reads description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7707111' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-osd-ec-partial-reads ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 1 ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd memory target: 1610612736 osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request sha1: ae2481253f9b2fa3451954e615eedb67ebdd5314 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: debuginfo: true flavor: default sha1: ae2481253f9b2fa3451954e615eedb67ebdd5314 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-osd-ec-partial-reads sha1: ae2481253f9b2fa3451954e615eedb67ebdd5314 owner: scheduled_rzarzynski@teuthology priority: 85 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 8148 sha1: ae2481253f9b2fa3451954e615eedb67ebdd5314 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-osd-ec-partial-reads suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_ae2481253f9b2fa3451954e615eedb67ebdd5314/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: ae2481253f9b2fa3451954e615eedb67ebdd5314 targets: smithi117.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIsXkaK4XQPzB8+ZJdXcA3a5GBSnAV9wzg7BG9USRcj7bw0CjWp7FrSJylm7crDcKiGM5FFlopish4Bjir3K9qc= smithi178.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPHTLb3nCf+LZPtkT+i7iba/GITUHDkKt0Jf053EyLYISLrDwZ2shMTtn0NUH8ax5kcE9uVIvY6w5Ahszapyv4w= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 1ae7ad82388e92a475afff437d49054826c019a1 timestamp: 2024-05-15_11:53:37 tube: smithi user: rzarzynski verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-05-15T12:22:56.554 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_ae2481253f9b2fa3451954e615eedb67ebdd5314/qa; will attempt to use it 2024-05-15T12:22:56.555 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_ae2481253f9b2fa3451954e615eedb67ebdd5314/qa/tasks 2024-05-15T12:22:56.555 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-05-15T12:22:56.556 INFO:teuthology.task.internal:Checking packages... 2024-05-15T12:22:56.579 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'ae2481253f9b2fa3451954e615eedb67ebdd5314' 2024-05-15T12:22:56.579 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-15T12:22:56.580 INFO:teuthology.packaging:ref: None 2024-05-15T12:22:56.580 INFO:teuthology.packaging:tag: None 2024-05-15T12:22:56.580 INFO:teuthology.packaging:branch: wip-osd-ec-partial-reads 2024-05-15T12:22:56.580 INFO:teuthology.packaging:sha1: ae2481253f9b2fa3451954e615eedb67ebdd5314 2024-05-15T12:22:56.580 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-osd-ec-partial-reads 2024-05-15T12:22:56.798 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1882.gae248125 2024-05-15T12:22:56.799 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-05-15T12:22:56.806 INFO:teuthology.task.internal:no buildpackages task found 2024-05-15T12:22:56.806 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-05-15T12:22:56.811 INFO:teuthology.task.internal:Saving configuration 2024-05-15T12:22:56.825 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-05-15T12:22:56.831 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-15T12:22:56.853 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi117.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707111', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-15 12:17:45.404670', 'locked_by': 'scheduled_rzarzynski@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIsXkaK4XQPzB8+ZJdXcA3a5GBSnAV9wzg7BG9USRcj7bw0CjWp7FrSJylm7crDcKiGM5FFlopish4Bjir3K9qc='} 2024-05-15T12:22:56.874 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi178.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707111', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-15 12:17:45.403394', 'locked_by': 'scheduled_rzarzynski@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPHTLb3nCf+LZPtkT+i7iba/GITUHDkKt0Jf053EyLYISLrDwZ2shMTtn0NUH8ax5kcE9uVIvY6w5Ahszapyv4w='} 2024-05-15T12:22:56.874 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-05-15T12:22:56.880 INFO:teuthology.task.internal:roles: ubuntu@smithi117.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-05-15T12:22:56.880 INFO:teuthology.task.internal:roles: ubuntu@smithi178.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-05-15T12:22:56.880 INFO:teuthology.run_tasks:Running task console_log... 2024-05-15T12:22:56.952 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fbed6bcf160>, signals=[15]) 2024-05-15T12:22:56.953 INFO:teuthology.run_tasks:Running task internal.connect... 2024-05-15T12:22:56.957 INFO:teuthology.task.internal:Opening connections... 2024-05-15T12:22:56.957 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi117.front.sepia.ceph.com 2024-05-15T12:22:56.959 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:22:57.034 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi178.front.sepia.ceph.com 2024-05-15T12:22:57.034 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:22:57.110 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-05-15T12:22:57.117 DEBUG:teuthology.orchestra.run.smithi117:> uname -m 2024-05-15T12:22:57.135 INFO:teuthology.orchestra.run.smithi117.stdout:x86_64 2024-05-15T12:22:57.135 DEBUG:teuthology.orchestra.run.smithi117:> cat /etc/os-release 2024-05-15T12:22:57.192 INFO:teuthology.orchestra.run.smithi117.stdout:NAME="CentOS Stream" 2024-05-15T12:22:57.192 INFO:teuthology.orchestra.run.smithi117.stdout:VERSION="9" 2024-05-15T12:22:57.192 INFO:teuthology.orchestra.run.smithi117.stdout:ID="centos" 2024-05-15T12:22:57.192 INFO:teuthology.orchestra.run.smithi117.stdout:ID_LIKE="rhel fedora" 2024-05-15T12:22:57.192 INFO:teuthology.orchestra.run.smithi117.stdout:VERSION_ID="9" 2024-05-15T12:22:57.192 INFO:teuthology.orchestra.run.smithi117.stdout:PLATFORM_ID="platform:el9" 2024-05-15T12:22:57.192 INFO:teuthology.orchestra.run.smithi117.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-15T12:22:57.192 INFO:teuthology.orchestra.run.smithi117.stdout:ANSI_COLOR="0;31" 2024-05-15T12:22:57.193 INFO:teuthology.orchestra.run.smithi117.stdout:LOGO="fedora-logo-icon" 2024-05-15T12:22:57.193 INFO:teuthology.orchestra.run.smithi117.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-15T12:22:57.193 INFO:teuthology.orchestra.run.smithi117.stdout:HOME_URL="https://centos.org/" 2024-05-15T12:22:57.193 INFO:teuthology.orchestra.run.smithi117.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-15T12:22:57.193 INFO:teuthology.orchestra.run.smithi117.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-15T12:22:57.193 INFO:teuthology.orchestra.run.smithi117.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-15T12:22:57.193 INFO:teuthology.lock.ops:Updating smithi117.front.sepia.ceph.com on lock server 2024-05-15T12:22:57.222 DEBUG:teuthology.orchestra.run.smithi178:> uname -m 2024-05-15T12:22:57.240 INFO:teuthology.orchestra.run.smithi178.stdout:x86_64 2024-05-15T12:22:57.240 DEBUG:teuthology.orchestra.run.smithi178:> cat /etc/os-release 2024-05-15T12:22:57.297 INFO:teuthology.orchestra.run.smithi178.stdout:NAME="CentOS Stream" 2024-05-15T12:22:57.297 INFO:teuthology.orchestra.run.smithi178.stdout:VERSION="9" 2024-05-15T12:22:57.297 INFO:teuthology.orchestra.run.smithi178.stdout:ID="centos" 2024-05-15T12:22:57.297 INFO:teuthology.orchestra.run.smithi178.stdout:ID_LIKE="rhel fedora" 2024-05-15T12:22:57.298 INFO:teuthology.orchestra.run.smithi178.stdout:VERSION_ID="9" 2024-05-15T12:22:57.298 INFO:teuthology.orchestra.run.smithi178.stdout:PLATFORM_ID="platform:el9" 2024-05-15T12:22:57.298 INFO:teuthology.orchestra.run.smithi178.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-15T12:22:57.298 INFO:teuthology.orchestra.run.smithi178.stdout:ANSI_COLOR="0;31" 2024-05-15T12:22:57.298 INFO:teuthology.orchestra.run.smithi178.stdout:LOGO="fedora-logo-icon" 2024-05-15T12:22:57.298 INFO:teuthology.orchestra.run.smithi178.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-15T12:22:57.298 INFO:teuthology.orchestra.run.smithi178.stdout:HOME_URL="https://centos.org/" 2024-05-15T12:22:57.298 INFO:teuthology.orchestra.run.smithi178.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-15T12:22:57.298 INFO:teuthology.orchestra.run.smithi178.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-15T12:22:57.298 INFO:teuthology.orchestra.run.smithi178.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-15T12:22:57.298 INFO:teuthology.lock.ops:Updating smithi178.front.sepia.ceph.com on lock server 2024-05-15T12:22:57.322 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-05-15T12:22:57.331 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-05-15T12:22:57.340 INFO:teuthology.task.internal:Checking for old test directory... 2024-05-15T12:22:57.341 DEBUG:teuthology.orchestra.run.smithi117:> test '!' -e /home/ubuntu/cephtest 2024-05-15T12:22:57.343 DEBUG:teuthology.orchestra.run.smithi178:> test '!' -e /home/ubuntu/cephtest 2024-05-15T12:22:57.360 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-05-15T12:22:57.365 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-05-15T12:22:57.365 DEBUG:teuthology.orchestra.run.smithi117:> test -z $(ls -A /var/lib/ceph) 2024-05-15T12:22:57.399 DEBUG:teuthology.orchestra.run.smithi178:> test -z $(ls -A /var/lib/ceph) 2024-05-15T12:22:57.494 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-05-15T12:22:57.548 INFO:teuthology.run_tasks:Running task kernel... 2024-05-15T12:22:57.563 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-05-15T12:22:57.563 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-05-15T12:22:57.563 DEBUG:teuthology.orchestra.run.smithi117:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-15T12:22:57.564 DEBUG:teuthology.orchestra.run.smithi178:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-15T12:22:57.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T12:22:57.580 DEBUG:teuthology.orchestra.run.smithi117:> uname -r 2024-05-15T12:22:57.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T12:22:57.581 DEBUG:teuthology.orchestra.run.smithi178:> uname -r 2024-05-15T12:22:57.636 INFO:teuthology.orchestra.run.smithi117.stdout:5.14.0-437.el9.x86_64 2024-05-15T12:22:57.637 INFO:teuthology.task.kernel:Running kernel on smithi117: 5.14.0-437.el9.x86_64 2024-05-15T12:22:57.637 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum install -y kernel 2024-05-15T12:22:57.638 INFO:teuthology.orchestra.run.smithi178.stdout:5.14.0-437.el9.x86_64 2024-05-15T12:22:57.638 INFO:teuthology.task.kernel:Running kernel on smithi178: 5.14.0-437.el9.x86_64 2024-05-15T12:22:57.638 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum install -y kernel 2024-05-15T12:22:59.807 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.1 MB 00:00 2024-05-15T12:23:01.370 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 9 - BaseOS 3.8 MB/s | 8.1 MB 00:02 2024-05-15T12:23:03.254 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 9 - AppStream 25 MB/s | 19 MB 00:00 2024-05-15T12:23:03.599 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - AppStream 7.1 MB/s | 19 MB 00:02 2024-05-15T12:23:07.613 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 9 - CRB 8.9 MB/s | 6.2 MB 00:00 2024-05-15T12:23:07.991 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - CRB 8.1 MB/s | 6.2 MB 00:00 2024-05-15T12:23:08.955 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 9 - Extras packages 94 kB/s | 16 kB 00:00 2024-05-15T12:23:09.992 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - Extras packages 29 kB/s | 16 kB 00:00 2024-05-15T12:23:10.507 INFO:teuthology.orchestra.run.smithi178.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-05-15T12:23:11.601 INFO:teuthology.orchestra.run.smithi117.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-05-15T12:23:15.682 INFO:teuthology.orchestra.run.smithi178.stdout:lab-extras 20 kB/s | 1.7 kB 00:00 2024-05-15T12:23:16.659 INFO:teuthology.orchestra.run.smithi117.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-05-15T12:23:17.671 INFO:teuthology.orchestra.run.smithi178.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-05-15T12:23:17.671 INFO:teuthology.orchestra.run.smithi178.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-05-15T12:23:17.765 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-05-15T12:23:17.772 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:23:17.772 INFO:teuthology.orchestra.run.smithi178.stdout: Package Architecture Version Repository Size 2024-05-15T12:23:17.773 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:23:17.773 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-05-15T12:23:17.773 INFO:teuthology.orchestra.run.smithi178.stdout: kernel x86_64 5.14.0-446.el9 baseos 550 k 2024-05-15T12:23:17.773 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-05-15T12:23:17.773 INFO:teuthology.orchestra.run.smithi178.stdout: kernel-core x86_64 5.14.0-446.el9 baseos 15 M 2024-05-15T12:23:17.773 INFO:teuthology.orchestra.run.smithi178.stdout: kernel-modules x86_64 5.14.0-446.el9 baseos 34 M 2024-05-15T12:23:17.773 INFO:teuthology.orchestra.run.smithi178.stdout: kernel-modules-core x86_64 5.14.0-446.el9 baseos 28 M 2024-05-15T12:23:17.773 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:23:17.773 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-05-15T12:23:17.773 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:23:17.773 INFO:teuthology.orchestra.run.smithi178.stdout:Install 4 Packages 2024-05-15T12:23:17.773 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:23:17.774 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 78 M 2024-05-15T12:23:17.774 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 123 M 2024-05-15T12:23:17.774 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-05-15T12:23:18.175 INFO:teuthology.orchestra.run.smithi178.stdout:(1/4): kernel-5.14.0-446.el9.x86_64.rpm 2.1 MB/s | 550 kB 00:00 2024-05-15T12:23:18.607 INFO:teuthology.orchestra.run.smithi117.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-05-15T12:23:18.607 INFO:teuthology.orchestra.run.smithi117.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-05-15T12:23:18.700 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-05-15T12:23:18.706 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:23:18.706 INFO:teuthology.orchestra.run.smithi117.stdout: Package Architecture Version Repository Size 2024-05-15T12:23:18.706 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:23:18.706 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-05-15T12:23:18.706 INFO:teuthology.orchestra.run.smithi117.stdout: kernel x86_64 5.14.0-446.el9 baseos 550 k 2024-05-15T12:23:18.706 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-05-15T12:23:18.707 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-core x86_64 5.14.0-446.el9 baseos 15 M 2024-05-15T12:23:18.707 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-modules x86_64 5.14.0-446.el9 baseos 34 M 2024-05-15T12:23:18.707 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-modules-core x86_64 5.14.0-446.el9 baseos 28 M 2024-05-15T12:23:18.707 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:23:18.707 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-05-15T12:23:18.707 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:23:18.707 INFO:teuthology.orchestra.run.smithi117.stdout:Install 4 Packages 2024-05-15T12:23:18.707 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:23:18.707 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 78 M 2024-05-15T12:23:18.707 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 123 M 2024-05-15T12:23:18.707 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-05-15T12:23:18.733 INFO:teuthology.orchestra.run.smithi178.stdout:(2/4): kernel-core-5.14.0-446.el9.x86_64.rpm 19 MB/s | 15 MB 00:00 2024-05-15T12:23:19.550 INFO:teuthology.orchestra.run.smithi178.stdout:(3/4): kernel-modules-5.14.0-446.el9.x86_64.rpm 21 MB/s | 34 MB 00:01 2024-05-15T12:23:20.085 INFO:teuthology.orchestra.run.smithi117.stdout:(1/4): kernel-5.14.0-446.el9.x86_64.rpm 412 kB/s | 550 kB 00:01 2024-05-15T12:23:20.234 INFO:teuthology.orchestra.run.smithi178.stdout:(4/4): kernel-modules-core-5.14.0-446.el9.x86_6 14 MB/s | 28 MB 00:02 2024-05-15T12:23:20.234 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:23:20.234 INFO:teuthology.orchestra.run.smithi178.stdout:Total 32 MB/s | 78 MB 00:02 2024-05-15T12:23:20.917 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-05-15T12:23:21.242 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-05-15T12:23:21.243 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-05-15T12:23:22.185 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-05-15T12:23:22.186 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-05-15T12:23:22.360 INFO:teuthology.orchestra.run.smithi117.stdout:(2/4): kernel-core-5.14.0-446.el9.x86_64.rpm 4.3 MB/s | 15 MB 00:03 2024-05-15T12:23:24.251 INFO:teuthology.orchestra.run.smithi117.stdout:(3/4): kernel-modules-5.14.0-446.el9.x86_64.rpm 6.2 MB/s | 34 MB 00:05 2024-05-15T12:23:24.594 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-05-15T12:23:24.760 INFO:teuthology.orchestra.run.smithi117.stdout:(4/4): kernel-modules-core-5.14.0-446.el9.x86_6 6.0 MB/s | 28 MB 00:04 2024-05-15T12:23:24.760 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:23:24.760 INFO:teuthology.orchestra.run.smithi117.stdout:Total 13 MB/s | 78 MB 00:06 2024-05-15T12:23:25.246 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : kernel-modules-core-5.14.0-446.el9.x86_64 1/4 2024-05-15T12:23:25.325 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : kernel-core-5.14.0-446.el9.x86_64 2/4 2024-05-15T12:23:25.438 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-05-15T12:23:25.755 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-05-15T12:23:25.756 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-05-15T12:23:26.684 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-05-15T12:23:26.684 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-05-15T12:23:27.983 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: kernel-core-5.14.0-446.el9.x86_64 2/4 2024-05-15T12:23:28.127 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : kernel-modules-5.14.0-446.el9.x86_64 3/4 2024-05-15T12:23:29.082 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-05-15T12:23:29.706 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : kernel-modules-core-5.14.0-446.el9.x86_64 1/4 2024-05-15T12:23:29.794 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : kernel-core-5.14.0-446.el9.x86_64 2/4 2024-05-15T12:23:32.467 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-core-5.14.0-446.el9.x86_64 2/4 2024-05-15T12:23:32.611 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : kernel-modules-5.14.0-446.el9.x86_64 3/4 2024-05-15T12:23:32.821 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: kernel-modules-5.14.0-446.el9.x86_64 3/4 2024-05-15T12:23:33.054 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : kernel-5.14.0-446.el9.x86_64 4/4 2024-05-15T12:23:37.416 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-modules-5.14.0-446.el9.x86_64 3/4 2024-05-15T12:23:37.464 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : kernel-5.14.0-446.el9.x86_64 4/4 2024-05-15T12:23:37.781 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: kernel-modules-core-5.14.0-446.el9.x86_64 4/4 2024-05-15T12:23:42.236 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-modules-core-5.14.0-446.el9.x86_64 4/4 2024-05-15T12:24:07.325 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: kernel-core-5.14.0-446.el9.x86_64 4/4 2024-05-15T12:24:07.364 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: kernel-modules-5.14.0-446.el9.x86_64 4/4 2024-05-15T12:24:07.747 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: kernel-5.14.0-446.el9.x86_64 4/4 2024-05-15T12:24:07.747 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : kernel-5.14.0-446.el9.x86_64 1/4 2024-05-15T12:24:07.747 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : kernel-core-5.14.0-446.el9.x86_64 2/4 2024-05-15T12:24:07.747 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : kernel-modules-5.14.0-446.el9.x86_64 3/4 2024-05-15T12:24:08.572 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : kernel-modules-core-5.14.0-446.el9.x86_64 4/4 2024-05-15T12:24:08.573 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:24:08.573 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-05-15T12:24:08.573 INFO:teuthology.orchestra.run.smithi178.stdout: kernel-5.14.0-446.el9.x86_64 2024-05-15T12:24:08.573 INFO:teuthology.orchestra.run.smithi178.stdout: kernel-core-5.14.0-446.el9.x86_64 2024-05-15T12:24:08.573 INFO:teuthology.orchestra.run.smithi178.stdout: kernel-modules-5.14.0-446.el9.x86_64 2024-05-15T12:24:08.573 INFO:teuthology.orchestra.run.smithi178.stdout: kernel-modules-core-5.14.0-446.el9.x86_64 2024-05-15T12:24:08.573 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:24:08.573 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-05-15T12:24:08.915 DEBUG:teuthology.orchestra.run.smithi178:> rpm -q kernel | sort -rV | head -n 1 2024-05-15T12:24:08.996 INFO:teuthology.orchestra.run.smithi178.stdout:kernel-5.14.0-446.el9.x86_64 2024-05-15T12:24:08.996 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-446.el9.x86_64 2024-05-15T12:24:08.997 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-446.el9.x86_64 2024-05-15T12:24:08.997 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-05-15T12:24:08.997 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-05-15T12:24:08.997 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-05-15T12:24:08.997 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi178.front.sepia.ceph.com, path=None, version=distro) 2024-05-15T12:24:08.997 DEBUG:teuthology.orchestra.run.smithi178:> rpm -q kernel | sort -rV | head -n 1 2024-05-15T12:24:09.047 INFO:teuthology.orchestra.run.smithi178.stdout:kernel-5.14.0-446.el9.x86_64 2024-05-15T12:24:09.047 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-446.el9.x86_64 2024-05-15T12:24:09.047 DEBUG:teuthology.orchestra.run.smithi178:> sudo rpm -qi grub2-tools 2024-05-15T12:24:09.079 INFO:teuthology.orchestra.run.smithi178.stdout:Name : grub2-tools 2024-05-15T12:24:09.079 INFO:teuthology.orchestra.run.smithi178.stdout:Epoch : 1 2024-05-15T12:24:09.079 INFO:teuthology.orchestra.run.smithi178.stdout:Version : 2.06 2024-05-15T12:24:09.079 INFO:teuthology.orchestra.run.smithi178.stdout:Release : 68.el9 2024-05-15T12:24:09.079 INFO:teuthology.orchestra.run.smithi178.stdout:Architecture: x86_64 2024-05-15T12:24:09.079 INFO:teuthology.orchestra.run.smithi178.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-05-15T12:24:09.079 INFO:teuthology.orchestra.run.smithi178.stdout:Group : Unspecified 2024-05-15T12:24:09.079 INFO:teuthology.orchestra.run.smithi178.stdout:Size : 8264254 2024-05-15T12:24:09.079 INFO:teuthology.orchestra.run.smithi178.stdout:License : GPLv3+ 2024-05-15T12:24:09.079 INFO:teuthology.orchestra.run.smithi178.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-05-15T12:24:09.079 INFO:teuthology.orchestra.run.smithi178.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-05-15T12:24:09.079 INFO:teuthology.orchestra.run.smithi178.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-05-15T12:24:09.080 INFO:teuthology.orchestra.run.smithi178.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-05-15T12:24:09.080 INFO:teuthology.orchestra.run.smithi178.stdout:Packager : builder@centos.org 2024-05-15T12:24:09.080 INFO:teuthology.orchestra.run.smithi178.stdout:Vendor : CentOS 2024-05-15T12:24:09.080 INFO:teuthology.orchestra.run.smithi178.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-15T12:24:09.080 INFO:teuthology.orchestra.run.smithi178.stdout:Summary : Support tools for GRUB. 2024-05-15T12:24:09.080 INFO:teuthology.orchestra.run.smithi178.stdout:Description : 2024-05-15T12:24:09.080 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:24:09.080 INFO:teuthology.orchestra.run.smithi178.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-15T12:24:09.080 INFO:teuthology.orchestra.run.smithi178.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-15T12:24:09.080 INFO:teuthology.orchestra.run.smithi178.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-15T12:24:09.080 INFO:teuthology.orchestra.run.smithi178.stdout:hardware devices. 2024-05-15T12:24:09.080 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:24:09.080 INFO:teuthology.orchestra.run.smithi178.stdout:This subpackage provides tools for support of all platforms. 2024-05-15T12:24:09.081 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-15T12:24:09.081 INFO:teuthology.task.kernel:Updating grub on smithi178 to boot 5.14.0-446.el9.x86_64 2024-05-15T12:24:09.081 DEBUG:teuthology.orchestra.run.smithi178:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-15T12:24:09.682 INFO:teuthology.orchestra.run.smithi178.stderr:Generating grub configuration file ... 2024-05-15T12:24:11.077 INFO:teuthology.orchestra.run.smithi178.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-05-15T12:24:11.108 INFO:teuthology.orchestra.run.smithi178.stderr:done 2024-05-15T12:24:11.115 DEBUG:teuthology.orchestra.run.smithi178:> mktemp 2024-05-15T12:24:11.130 INFO:teuthology.orchestra.run.smithi178.stdout:/tmp/tmp.JN5vQ5KPVL 2024-05-15T12:24:11.130 DEBUG:teuthology.orchestra.run.smithi178:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.JN5vQ5KPVL 2024-05-15T12:24:11.196 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod 0666 /tmp/tmp.JN5vQ5KPVL 2024-05-15T12:24:11.336 DEBUG:teuthology.orchestra.remote:smithi178:/tmp/tmp.JN5vQ5KPVL is 6KB 2024-05-15T12:24:11.384 DEBUG:teuthology.orchestra.run.smithi178:> rm -fr /tmp/tmp.JN5vQ5KPVL 2024-05-15T12:24:11.398 DEBUG:teuthology.orchestra.run.smithi178:> sudo /bin/ls /boot/loader/entries || true 2024-05-15T12:24:11.462 INFO:teuthology.orchestra.run.smithi178.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-05-15T12:24:11.462 INFO:teuthology.orchestra.run.smithi178.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-05-15T12:24:11.462 INFO:teuthology.orchestra.run.smithi178.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-05-15T12:24:11.462 INFO:teuthology.orchestra.run.smithi178.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-446.el9.x86_64.conf 2024-05-15T12:24:11.463 DEBUG:teuthology.orchestra.run.smithi178:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-446.el9.x86_64 2024-05-15T12:24:11.542 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-core-5.14.0-446.el9.x86_64 4/4 2024-05-15T12:24:11.581 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-modules-5.14.0-446.el9.x86_64 4/4 2024-05-15T12:24:11.614 DEBUG:teuthology.orchestra.run.smithi178:> sudo shutdown -r now 2024-05-15T12:24:11.956 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-5.14.0-446.el9.x86_64 4/4 2024-05-15T12:24:11.956 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : kernel-5.14.0-446.el9.x86_64 1/4 2024-05-15T12:24:11.956 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : kernel-core-5.14.0-446.el9.x86_64 2/4 2024-05-15T12:24:11.956 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : kernel-modules-5.14.0-446.el9.x86_64 3/4 2024-05-15T12:24:12.739 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : kernel-modules-core-5.14.0-446.el9.x86_64 4/4 2024-05-15T12:24:12.739 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:24:12.740 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-05-15T12:24:12.740 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-5.14.0-446.el9.x86_64 2024-05-15T12:24:12.740 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-core-5.14.0-446.el9.x86_64 2024-05-15T12:24:12.740 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-modules-5.14.0-446.el9.x86_64 2024-05-15T12:24:12.740 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-modules-core-5.14.0-446.el9.x86_64 2024-05-15T12:24:12.740 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:24:12.740 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-05-15T12:24:13.065 DEBUG:teuthology.orchestra.run.smithi117:> rpm -q kernel | sort -rV | head -n 1 2024-05-15T12:24:13.143 INFO:teuthology.orchestra.run.smithi117.stdout:kernel-5.14.0-446.el9.x86_64 2024-05-15T12:24:13.143 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-446.el9.x86_64 2024-05-15T12:24:13.143 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-446.el9.x86_64 2024-05-15T12:24:13.144 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-05-15T12:24:13.144 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-05-15T12:24:13.144 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-05-15T12:24:13.144 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi117.front.sepia.ceph.com, path=None, version=distro) 2024-05-15T12:24:13.144 DEBUG:teuthology.orchestra.run.smithi117:> rpm -q kernel | sort -rV | head -n 1 2024-05-15T12:24:13.194 INFO:teuthology.orchestra.run.smithi117.stdout:kernel-5.14.0-446.el9.x86_64 2024-05-15T12:24:13.194 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-446.el9.x86_64 2024-05-15T12:24:13.194 DEBUG:teuthology.orchestra.run.smithi117:> sudo rpm -qi grub2-tools 2024-05-15T12:24:13.225 INFO:teuthology.orchestra.run.smithi117.stdout:Name : grub2-tools 2024-05-15T12:24:13.225 INFO:teuthology.orchestra.run.smithi117.stdout:Epoch : 1 2024-05-15T12:24:13.225 INFO:teuthology.orchestra.run.smithi117.stdout:Version : 2.06 2024-05-15T12:24:13.225 INFO:teuthology.orchestra.run.smithi117.stdout:Release : 68.el9 2024-05-15T12:24:13.225 INFO:teuthology.orchestra.run.smithi117.stdout:Architecture: x86_64 2024-05-15T12:24:13.225 INFO:teuthology.orchestra.run.smithi117.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-05-15T12:24:13.225 INFO:teuthology.orchestra.run.smithi117.stdout:Group : Unspecified 2024-05-15T12:24:13.225 INFO:teuthology.orchestra.run.smithi117.stdout:Size : 8264254 2024-05-15T12:24:13.225 INFO:teuthology.orchestra.run.smithi117.stdout:License : GPLv3+ 2024-05-15T12:24:13.225 INFO:teuthology.orchestra.run.smithi117.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-05-15T12:24:13.226 INFO:teuthology.orchestra.run.smithi117.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-05-15T12:24:13.226 INFO:teuthology.orchestra.run.smithi117.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-05-15T12:24:13.226 INFO:teuthology.orchestra.run.smithi117.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-05-15T12:24:13.226 INFO:teuthology.orchestra.run.smithi117.stdout:Packager : builder@centos.org 2024-05-15T12:24:13.226 INFO:teuthology.orchestra.run.smithi117.stdout:Vendor : CentOS 2024-05-15T12:24:13.226 INFO:teuthology.orchestra.run.smithi117.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-15T12:24:13.226 INFO:teuthology.orchestra.run.smithi117.stdout:Summary : Support tools for GRUB. 2024-05-15T12:24:13.226 INFO:teuthology.orchestra.run.smithi117.stdout:Description : 2024-05-15T12:24:13.226 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:24:13.226 INFO:teuthology.orchestra.run.smithi117.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-15T12:24:13.226 INFO:teuthology.orchestra.run.smithi117.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-15T12:24:13.226 INFO:teuthology.orchestra.run.smithi117.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-15T12:24:13.226 INFO:teuthology.orchestra.run.smithi117.stdout:hardware devices. 2024-05-15T12:24:13.226 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:24:13.227 INFO:teuthology.orchestra.run.smithi117.stdout:This subpackage provides tools for support of all platforms. 2024-05-15T12:24:13.227 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-15T12:24:13.227 INFO:teuthology.task.kernel:Updating grub on smithi117 to boot 5.14.0-446.el9.x86_64 2024-05-15T12:24:13.228 DEBUG:teuthology.orchestra.run.smithi117:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-15T12:24:13.989 INFO:teuthology.orchestra.run.smithi117.stderr:Generating grub configuration file ... 2024-05-15T12:24:15.305 INFO:teuthology.orchestra.run.smithi117.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-05-15T12:24:15.339 INFO:teuthology.orchestra.run.smithi117.stderr:done 2024-05-15T12:24:15.341 DEBUG:teuthology.orchestra.run.smithi117:> mktemp 2024-05-15T12:24:15.396 INFO:teuthology.orchestra.run.smithi117.stdout:/tmp/tmp.SQV8jRvJCH 2024-05-15T12:24:15.396 DEBUG:teuthology.orchestra.run.smithi117:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.SQV8jRvJCH 2024-05-15T12:24:15.461 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0666 /tmp/tmp.SQV8jRvJCH 2024-05-15T12:24:15.594 DEBUG:teuthology.orchestra.remote:smithi117:/tmp/tmp.SQV8jRvJCH is 6KB 2024-05-15T12:24:15.643 DEBUG:teuthology.orchestra.run.smithi117:> rm -fr /tmp/tmp.SQV8jRvJCH 2024-05-15T12:24:15.657 DEBUG:teuthology.orchestra.run.smithi117:> sudo /bin/ls /boot/loader/entries || true 2024-05-15T12:24:15.718 INFO:teuthology.orchestra.run.smithi117.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-05-15T12:24:15.719 INFO:teuthology.orchestra.run.smithi117.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-05-15T12:24:15.719 INFO:teuthology.orchestra.run.smithi117.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-05-15T12:24:15.719 INFO:teuthology.orchestra.run.smithi117.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-446.el9.x86_64.conf 2024-05-15T12:24:15.720 DEBUG:teuthology.orchestra.run.smithi117:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-446.el9.x86_64 2024-05-15T12:24:15.897 DEBUG:teuthology.orchestra.run.smithi117:> sudo shutdown -r now 2024-05-15T12:24:41.628 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-05-15T12:24:41.629 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-05-15T12:24:41.630 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:24:45.902 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-05-15T12:24:45.902 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-05-15T12:24:45.903 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:25:00.190 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.178 2024-05-15T12:25:04.286 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.117 2024-05-15T12:25:09.196 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-05-15T12:25:09.197 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:25:12.258 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.178 2024-05-15T12:25:13.288 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-05-15T12:25:13.288 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:25:16.350 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.117 2024-05-15T12:25:24.265 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-05-15T12:25:24.266 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:25:28.354 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-05-15T12:25:28.355 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:25:46.782 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.117 2024-05-15T12:26:01.794 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-05-15T12:26:01.795 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:26:02.145 DEBUG:teuthology.orchestra.run.smithi117:> true 2024-05-15T12:26:02.503 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-05-15T12:26:02.504 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-446.el9.x86_64"... 2024-05-15T12:26:02.504 DEBUG:teuthology.orchestra.run.smithi117:> uname -r 2024-05-15T12:26:02.558 INFO:teuthology.orchestra.run.smithi117.stdout:5.14.0-446.el9.x86_64 2024-05-15T12:26:02.559 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-446.el9.x86_64 vs 5.14.0-446.el9.x86_64 2024-05-15T12:26:02.559 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-15T12:26:02.559 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-15T12:26:03.560 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-05-15T12:26:03.560 DEBUG:teuthology.orchestra.run.smithi117:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-15T12:26:03.629 INFO:teuthology.orchestra.run.smithi117.stdout:ttyS1 2024-05-15T12:26:03.666 DEBUG:teuthology.parallel:result is None 2024-05-15T12:26:24.269 DEBUG:teuthology.orchestra.remote:timed out 2024-05-15T12:26:39.271 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-05-15T12:26:39.271 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:26:39.667 DEBUG:teuthology.orchestra.run.smithi178:> true 2024-05-15T12:26:40.064 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-05-15T12:26:40.064 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-446.el9.x86_64"... 2024-05-15T12:26:40.064 DEBUG:teuthology.orchestra.run.smithi178:> uname -r 2024-05-15T12:26:40.080 INFO:teuthology.orchestra.run.smithi178.stdout:5.14.0-446.el9.x86_64 2024-05-15T12:26:40.081 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-446.el9.x86_64 vs 5.14.0-446.el9.x86_64 2024-05-15T12:26:40.081 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-15T12:26:40.081 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-15T12:26:41.081 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-05-15T12:26:41.082 DEBUG:teuthology.orchestra.run.smithi178:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-15T12:26:41.146 INFO:teuthology.orchestra.run.smithi178.stdout:ttyS1 2024-05-15T12:26:41.183 DEBUG:teuthology.parallel:result is None 2024-05-15T12:26:41.184 INFO:teuthology.run_tasks:Running task internal.base... 2024-05-15T12:26:41.194 INFO:teuthology.task.internal:Creating test directory... 2024-05-15T12:26:41.194 DEBUG:teuthology.orchestra.run.smithi117:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-15T12:26:41.197 DEBUG:teuthology.orchestra.run.smithi178:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-15T12:26:41.216 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-05-15T12:26:41.223 INFO:teuthology.run_tasks:Running task internal.archive... 2024-05-15T12:26:41.229 INFO:teuthology.task.internal:Creating archive directory... 2024-05-15T12:26:41.230 DEBUG:teuthology.orchestra.run.smithi117:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-15T12:26:41.258 DEBUG:teuthology.orchestra.run.smithi178:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-15T12:26:41.349 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-05-15T12:26:41.355 INFO:teuthology.task.internal:Enabling coredump saving... 2024-05-15T12:26:41.356 DEBUG:teuthology.orchestra.run.smithi117:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-15T12:26:41.358 DEBUG:teuthology.orchestra.run.smithi178:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-15T12:26:41.387 INFO:teuthology.orchestra.run.smithi117.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:26:41.396 INFO:teuthology.orchestra.run.smithi178.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:26:41.398 INFO:teuthology.orchestra.run.smithi117.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:26:41.407 INFO:teuthology.orchestra.run.smithi178.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:26:41.409 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-05-15T12:26:41.416 INFO:teuthology.task.internal:Configuring sudo... 2024-05-15T12:26:41.416 DEBUG:teuthology.orchestra.run.smithi117:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-15T12:26:41.443 DEBUG:teuthology.orchestra.run.smithi178:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-15T12:26:41.478 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-05-15T12:26:41.487 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-05-15T12:26:41.487 DEBUG:teuthology.orchestra.run.smithi117:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-15T12:26:41.511 DEBUG:teuthology.orchestra.run.smithi178:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-15T12:26:41.535 DEBUG:teuthology.orchestra.run.smithi117:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-15T12:26:41.610 DEBUG:teuthology.orchestra.run.smithi117:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-15T12:26:41.661 DEBUG:teuthology.orchestra.run.smithi117:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-15T12:26:41.743 DEBUG:teuthology.orchestra.run.smithi117:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-15T12:26:41.787 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-05-15T12:26:41.787 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-15T12:26:41.854 DEBUG:teuthology.orchestra.run.smithi178:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-15T12:26:41.902 DEBUG:teuthology.orchestra.run.smithi178:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-15T12:26:41.955 DEBUG:teuthology.orchestra.run.smithi178:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-15T12:26:42.037 DEBUG:teuthology.orchestra.run.smithi178:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-15T12:26:42.085 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-05-15T12:26:42.085 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-15T12:26:42.153 DEBUG:teuthology.orchestra.run.smithi117:> sudo service rsyslog restart 2024-05-15T12:26:42.156 DEBUG:teuthology.orchestra.run.smithi178:> sudo service rsyslog restart 2024-05-15T12:26:42.210 INFO:teuthology.orchestra.run.smithi117.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-15T12:26:42.241 INFO:teuthology.orchestra.run.smithi178.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-15T12:26:42.679 INFO:teuthology.run_tasks:Running task internal.timer... 2024-05-15T12:26:42.687 INFO:teuthology.task.internal:Starting timer... 2024-05-15T12:26:42.688 INFO:teuthology.run_tasks:Running task pcp... 2024-05-15T12:26:42.709 INFO:teuthology.run_tasks:Running task selinux... 2024-05-15T12:26:42.718 DEBUG:teuthology.orchestra.run.smithi117:> sudo service auditd rotate 2024-05-15T12:26:42.786 INFO:teuthology.orchestra.run.smithi117.stdout:Rotating logs: 2024-05-15T12:26:42.788 DEBUG:teuthology.orchestra.run.smithi178:> sudo service auditd rotate 2024-05-15T12:26:42.836 INFO:teuthology.orchestra.run.smithi178.stdout:Rotating logs: 2024-05-15T12:26:42.838 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-05-15T12:26:42.838 DEBUG:teuthology.orchestra.run.smithi117:> /usr/sbin/getenforce 2024-05-15T12:26:42.865 INFO:teuthology.orchestra.run.smithi117.stdout:Permissive 2024-05-15T12:26:42.866 DEBUG:teuthology.orchestra.run.smithi178:> /usr/sbin/getenforce 2024-05-15T12:26:42.942 INFO:teuthology.orchestra.run.smithi178.stdout:Permissive 2024-05-15T12:26:42.942 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi117.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi178.front.sepia.ceph.com': 'permissive'} 2024-05-15T12:26:42.942 DEBUG:teuthology.orchestra.run.smithi117:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-05-15T12:26:42.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T12:26:42.971 DEBUG:teuthology.orchestra.run.smithi178:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-05-15T12:26:43.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T12:26:43.000 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-05-15T12:26:43.000 DEBUG:teuthology.orchestra.run.smithi117:> sudo /usr/sbin/setenforce permissive 2024-05-15T12:26:43.043 DEBUG:teuthology.orchestra.run.smithi178:> sudo /usr/sbin/setenforce permissive 2024-05-15T12:26:43.071 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-05-15T12:26:43.083 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-05-15T12:26:43.099 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-05-15T12:26:43.100 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi117.front.sepia.ceph.com,smithi178.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-05-15T12:31:04.964 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi117.front.sepia.ceph.com'), Remote(name='ubuntu@smithi178.front.sepia.ceph.com')] 2024-05-15T12:31:04.965 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-05-15T12:31:04.966 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:31:05.041 DEBUG:teuthology.orchestra.run.smithi117:> true 2024-05-15T12:31:05.121 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-05-15T12:31:05.122 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-05-15T12:31:05.122 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:31:05.198 DEBUG:teuthology.orchestra.run.smithi178:> true 2024-05-15T12:31:05.274 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-05-15T12:31:05.274 INFO:teuthology.run_tasks:Running task clock... 2024-05-15T12:31:05.284 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-05-15T12:31:05.284 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-15T12:31:05.285 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-15T12:31:05.287 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-15T12:31:05.287 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-15T12:31:05.315 INFO:teuthology.orchestra.run.smithi117.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-15T12:31:05.330 INFO:teuthology.orchestra.run.smithi117.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-15T12:31:05.344 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-15T12:31:05.355 INFO:teuthology.orchestra.run.smithi117.stderr:sudo: ntpd: command not found 2024-05-15T12:31:05.358 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-15T12:31:05.366 INFO:teuthology.orchestra.run.smithi117.stdout:506 Cannot talk to daemon 2024-05-15T12:31:05.380 INFO:teuthology.orchestra.run.smithi117.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-15T12:31:05.382 INFO:teuthology.orchestra.run.smithi178.stderr:sudo: ntpd: command not found 2024-05-15T12:31:05.394 INFO:teuthology.orchestra.run.smithi178.stdout:506 Cannot talk to daemon 2024-05-15T12:31:05.395 INFO:teuthology.orchestra.run.smithi117.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-15T12:31:05.407 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-15T12:31:05.421 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-15T12:31:05.449 INFO:teuthology.orchestra.run.smithi117.stderr:bash: line 1: ntpq: command not found 2024-05-15T12:31:05.453 INFO:teuthology.orchestra.run.smithi117.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-15T12:31:05.453 INFO:teuthology.orchestra.run.smithi117.stdout:=============================================================================== 2024-05-15T12:31:05.453 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T12:31:05.453 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T12:31:05.453 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T12:31:05.453 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T12:31:05.475 INFO:teuthology.orchestra.run.smithi178.stderr:bash: line 1: ntpq: command not found 2024-05-15T12:31:05.479 INFO:teuthology.orchestra.run.smithi178.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-15T12:31:05.479 INFO:teuthology.orchestra.run.smithi178.stdout:=============================================================================== 2024-05-15T12:31:05.479 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T12:31:05.479 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T12:31:05.479 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T12:31:05.479 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T12:31:05.480 INFO:teuthology.run_tasks:Running task install... 2024-05-15T12:31:05.489 DEBUG:teuthology.task.install:project ceph 2024-05-15T12:31:05.489 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': 'ae2481253f9b2fa3451954e615eedb67ebdd5314'}} 2024-05-15T12:31:05.489 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': 'ae2481253f9b2fa3451954e615eedb67ebdd5314'} 2024-05-15T12:31:05.490 INFO:teuthology.task.install:Using flavor: default 2024-05-15T12:31:05.494 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-05-15T12:31:05.495 INFO:teuthology.task.install:extra packages: [] 2024-05-15T12:31:05.495 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ae2481253f9b2fa3451954e615eedb67ebdd5314', 'tag': None, 'wait_for_package': False} 2024-05-15T12:31:05.495 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=ae2481253f9b2fa3451954e615eedb67ebdd5314 2024-05-15T12:31:05.497 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ae2481253f9b2fa3451954e615eedb67ebdd5314', 'tag': None, 'wait_for_package': False} 2024-05-15T12:31:05.497 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=ae2481253f9b2fa3451954e615eedb67ebdd5314 2024-05-15T12:31:05.638 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/centos/9/flavors/default/ 2024-05-15T12:31:05.638 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1882.gae248125 2024-05-15T12:31:05.645 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/centos/9/flavors/default/ 2024-05-15T12:31:05.645 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1882.gae248125 2024-05-15T12:31:05.945 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-15T12:31:05.945 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-05-15T12:31:05.946 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-15T12:31:05.952 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-15T12:31:05.953 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-05-15T12:31:05.953 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-15T12:31:05.973 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-05-15T12:31:05.974 DEBUG:teuthology.orchestra.run.smithi117:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/ae2481253f9b2fa3451954e615eedb67ebdd5314/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-15T12:31:05.979 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-05-15T12:31:05.979 DEBUG:teuthology.orchestra.run.smithi178:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/ae2481253f9b2fa3451954e615eedb67ebdd5314/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-15T12:31:06.039 DEBUG:teuthology.orchestra.run.smithi117:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-15T12:31:06.045 DEBUG:teuthology.orchestra.run.smithi178:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-15T12:31:06.114 DEBUG:teuthology.orchestra.run.smithi117:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-15T12:31:06.121 DEBUG:teuthology.orchestra.run.smithi178:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-15T12:31:06.179 INFO:teuthology.orchestra.run.smithi117.stdout:check_obsoletes = 1 2024-05-15T12:31:06.180 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum clean all 2024-05-15T12:31:06.185 INFO:teuthology.orchestra.run.smithi178.stdout:check_obsoletes = 1 2024-05-15T12:31:06.187 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum clean all 2024-05-15T12:31:06.410 INFO:teuthology.orchestra.run.smithi117.stdout:57 files removed 2024-05-15T12:31:06.428 INFO:teuthology.orchestra.run.smithi178.stdout:57 files removed 2024-05-15T12:31:06.437 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-radosgw 2024-05-15T12:31:06.455 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-radosgw 2024-05-15T12:31:07.415 INFO:teuthology.orchestra.run.smithi117.stdout:ceph packages for x86_64 116 kB/s | 82 kB 00:00 2024-05-15T12:31:07.518 INFO:teuthology.orchestra.run.smithi178.stdout:ceph packages for x86_64 103 kB/s | 82 kB 00:00 2024-05-15T12:31:08.107 INFO:teuthology.orchestra.run.smithi117.stdout:ceph noarch packages 16 kB/s | 10 kB 00:00 2024-05-15T12:31:08.177 INFO:teuthology.orchestra.run.smithi178.stdout:ceph noarch packages 17 kB/s | 10 kB 00:00 2024-05-15T12:31:08.533 INFO:teuthology.orchestra.run.smithi117.stdout:ceph source packages 5.0 kB/s | 1.9 kB 00:00 2024-05-15T12:31:08.752 INFO:teuthology.orchestra.run.smithi178.stdout:ceph source packages 3.6 kB/s | 1.9 kB 00:00 2024-05-15T12:31:09.400 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - BaseOS 9.8 MB/s | 8.1 MB 00:00 2024-05-15T12:31:09.978 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 9 - BaseOS 6.8 MB/s | 8.1 MB 00:01 2024-05-15T12:31:11.100 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - AppStream 32 MB/s | 19 MB 00:00 2024-05-15T12:31:13.445 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 9 - AppStream 8.0 MB/s | 19 MB 00:02 2024-05-15T12:31:15.351 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - CRB 8.8 MB/s | 6.2 MB 00:00 2024-05-15T12:31:16.985 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - Extras packages 39 kB/s | 16 kB 00:00 2024-05-15T12:31:18.029 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 9 - CRB 6.2 MB/s | 6.2 MB 00:00 2024-05-15T12:31:18.431 INFO:teuthology.orchestra.run.smithi117.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-05-15T12:31:19.521 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 9 - Extras packages 52 kB/s | 16 kB 00:00 2024-05-15T12:31:20.315 INFO:teuthology.orchestra.run.smithi178.stdout:Extra Packages for Enterprise Linux 32 MB/s | 21 MB 00:00 2024-05-15T12:31:23.586 INFO:teuthology.orchestra.run.smithi117.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2024-05-15T12:31:25.606 INFO:teuthology.orchestra.run.smithi178.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-05-15T12:31:25.943 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-05-15T12:31:25.953 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:31:25.953 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-05-15T12:31:25.954 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:31:25.954 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-05-15T12:31:25.954 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-radosgw x86_64 2:19.0.0-1882.gae248125.el9 ceph 9.8 M 2024-05-15T12:31:25.954 INFO:teuthology.orchestra.run.smithi117.stdout:Upgrading: 2024-05-15T12:31:25.954 INFO:teuthology.orchestra.run.smithi117.stdout: librados2 x86_64 2:19.0.0-1882.gae248125.el9 ceph 3.4 M 2024-05-15T12:31:25.954 INFO:teuthology.orchestra.run.smithi117.stdout: librbd1 x86_64 2:19.0.0-1882.gae248125.el9 ceph 3.1 M 2024-05-15T12:31:25.954 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-05-15T12:31:25.954 INFO:teuthology.orchestra.run.smithi117.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-15T12:31:25.954 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-base x86_64 2:19.0.0-1882.gae248125.el9 ceph 5.4 M 2024-05-15T12:31:25.954 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-common x86_64 2:19.0.0-1882.gae248125.el9 ceph 20 M 2024-05-15T12:31:25.954 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-selinux x86_64 2:19.0.0-1882.gae248125.el9 ceph 25 k 2024-05-15T12:31:25.954 INFO:teuthology.orchestra.run.smithi117.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-15T12:31:25.954 INFO:teuthology.orchestra.run.smithi117.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-15T12:31:25.955 INFO:teuthology.orchestra.run.smithi117.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-15T12:31:25.955 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs2 x86_64 2:19.0.0-1882.gae248125.el9 ceph 744 k 2024-05-15T12:31:25.955 INFO:teuthology.orchestra.run.smithi117.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-15T12:31:25.955 INFO:teuthology.orchestra.run.smithi117.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-15T12:31:25.955 INFO:teuthology.orchestra.run.smithi117.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-15T12:31:25.955 INFO:teuthology.orchestra.run.smithi117.stdout: libradosstriper1 x86_64 2:19.0.0-1882.gae248125.el9 ceph 501 k 2024-05-15T12:31:25.955 INFO:teuthology.orchestra.run.smithi117.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-15T12:31:25.955 INFO:teuthology.orchestra.run.smithi117.stdout: librgw2 x86_64 2:19.0.0-1882.gae248125.el9 ceph 5.1 M 2024-05-15T12:31:25.955 INFO:teuthology.orchestra.run.smithi117.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-15T12:31:25.955 INFO:teuthology.orchestra.run.smithi117.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-15T12:31:25.955 INFO:teuthology.orchestra.run.smithi117.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-15T12:31:25.955 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ceph-argparse x86_64 2:19.0.0-1882.gae248125.el9 ceph 45 k 2024-05-15T12:31:25.955 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ceph-common x86_64 2:19.0.0-1882.gae248125.el9 ceph 131 k 2024-05-15T12:31:25.955 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cephfs x86_64 2:19.0.0-1882.gae248125.el9 ceph 162 k 2024-05-15T12:31:25.955 INFO:teuthology.orchestra.run.smithi117.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-15T12:31:25.955 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rados x86_64 2:19.0.0-1882.gae248125.el9 ceph 321 k 2024-05-15T12:31:25.956 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rbd x86_64 2:19.0.0-1882.gae248125.el9 ceph 298 k 2024-05-15T12:31:25.956 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rgw x86_64 2:19.0.0-1882.gae248125.el9 ceph 99 k 2024-05-15T12:31:25.956 INFO:teuthology.orchestra.run.smithi117.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-15T12:31:25.956 INFO:teuthology.orchestra.run.smithi117.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-15T12:31:25.956 INFO:teuthology.orchestra.run.smithi117.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-15T12:31:25.956 INFO:teuthology.orchestra.run.smithi117.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-15T12:31:25.956 INFO:teuthology.orchestra.run.smithi117.stdout:Installing weak dependencies: 2024-05-15T12:31:25.956 INFO:teuthology.orchestra.run.smithi117.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-15T12:31:25.956 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:31:25.956 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-05-15T12:31:25.956 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:31:25.956 INFO:teuthology.orchestra.run.smithi117.stdout:Install 30 Packages 2024-05-15T12:31:25.956 INFO:teuthology.orchestra.run.smithi117.stdout:Upgrade 2 Packages 2024-05-15T12:31:25.956 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:31:25.957 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 58 M 2024-05-15T12:31:25.957 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-05-15T12:31:27.956 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-05-15T12:31:27.964 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:31:27.964 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-05-15T12:31:27.964 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:31:27.964 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-05-15T12:31:27.964 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-radosgw x86_64 2:19.0.0-1882.gae248125.el9 ceph 9.8 M 2024-05-15T12:31:27.964 INFO:teuthology.orchestra.run.smithi178.stdout:Upgrading: 2024-05-15T12:31:27.964 INFO:teuthology.orchestra.run.smithi178.stdout: librados2 x86_64 2:19.0.0-1882.gae248125.el9 ceph 3.4 M 2024-05-15T12:31:27.964 INFO:teuthology.orchestra.run.smithi178.stdout: librbd1 x86_64 2:19.0.0-1882.gae248125.el9 ceph 3.1 M 2024-05-15T12:31:27.964 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-05-15T12:31:27.964 INFO:teuthology.orchestra.run.smithi178.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-15T12:31:27.964 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-base x86_64 2:19.0.0-1882.gae248125.el9 ceph 5.4 M 2024-05-15T12:31:27.965 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-common x86_64 2:19.0.0-1882.gae248125.el9 ceph 20 M 2024-05-15T12:31:27.965 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-selinux x86_64 2:19.0.0-1882.gae248125.el9 ceph 25 k 2024-05-15T12:31:27.965 INFO:teuthology.orchestra.run.smithi178.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-15T12:31:27.965 INFO:teuthology.orchestra.run.smithi178.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-15T12:31:27.965 INFO:teuthology.orchestra.run.smithi178.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-15T12:31:27.965 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs2 x86_64 2:19.0.0-1882.gae248125.el9 ceph 744 k 2024-05-15T12:31:27.965 INFO:teuthology.orchestra.run.smithi178.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-15T12:31:27.965 INFO:teuthology.orchestra.run.smithi178.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-15T12:31:27.965 INFO:teuthology.orchestra.run.smithi178.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-15T12:31:27.965 INFO:teuthology.orchestra.run.smithi178.stdout: libradosstriper1 x86_64 2:19.0.0-1882.gae248125.el9 ceph 501 k 2024-05-15T12:31:27.965 INFO:teuthology.orchestra.run.smithi178.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-15T12:31:27.965 INFO:teuthology.orchestra.run.smithi178.stdout: librgw2 x86_64 2:19.0.0-1882.gae248125.el9 ceph 5.1 M 2024-05-15T12:31:27.965 INFO:teuthology.orchestra.run.smithi178.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-15T12:31:27.965 INFO:teuthology.orchestra.run.smithi178.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-15T12:31:27.965 INFO:teuthology.orchestra.run.smithi178.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-15T12:31:27.966 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-argparse x86_64 2:19.0.0-1882.gae248125.el9 ceph 45 k 2024-05-15T12:31:27.966 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-common x86_64 2:19.0.0-1882.gae248125.el9 ceph 131 k 2024-05-15T12:31:27.966 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cephfs x86_64 2:19.0.0-1882.gae248125.el9 ceph 162 k 2024-05-15T12:31:27.966 INFO:teuthology.orchestra.run.smithi178.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-15T12:31:27.966 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rados x86_64 2:19.0.0-1882.gae248125.el9 ceph 321 k 2024-05-15T12:31:27.966 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rbd x86_64 2:19.0.0-1882.gae248125.el9 ceph 298 k 2024-05-15T12:31:27.966 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rgw x86_64 2:19.0.0-1882.gae248125.el9 ceph 99 k 2024-05-15T12:31:27.966 INFO:teuthology.orchestra.run.smithi178.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-15T12:31:27.966 INFO:teuthology.orchestra.run.smithi178.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-15T12:31:27.966 INFO:teuthology.orchestra.run.smithi178.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-15T12:31:27.966 INFO:teuthology.orchestra.run.smithi178.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-15T12:31:27.966 INFO:teuthology.orchestra.run.smithi178.stdout:Installing weak dependencies: 2024-05-15T12:31:27.966 INFO:teuthology.orchestra.run.smithi178.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-15T12:31:27.966 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:31:27.966 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-05-15T12:31:27.967 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:31:27.967 INFO:teuthology.orchestra.run.smithi178.stdout:Install 30 Packages 2024-05-15T12:31:27.967 INFO:teuthology.orchestra.run.smithi178.stdout:Upgrade 2 Packages 2024-05-15T12:31:27.967 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:31:27.967 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 58 M 2024-05-15T12:31:27.967 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-05-15T12:31:28.387 INFO:teuthology.orchestra.run.smithi117.stdout:(1/32): ceph-base-19.0.0-1882.gae248125.el9.x86 2.7 MB/s | 5.4 MB 00:01 2024-05-15T12:31:28.487 INFO:teuthology.orchestra.run.smithi117.stdout:(2/32): ceph-selinux-19.0.0-1882.gae248125.el9. 252 kB/s | 25 kB 00:00 2024-05-15T12:31:28.854 INFO:teuthology.orchestra.run.smithi117.stdout:(3/32): libcephfs2-19.0.0-1882.gae248125.el9.x8 2.0 MB/s | 744 kB 00:00 2024-05-15T12:31:29.104 INFO:teuthology.orchestra.run.smithi117.stdout:(4/32): libradosstriper1-19.0.0-1882.gae248125. 2.0 MB/s | 501 kB 00:00 2024-05-15T12:31:29.204 INFO:teuthology.orchestra.run.smithi117.stdout:(5/32): ceph-radosgw-19.0.0-1882.gae248125.el9. 3.5 MB/s | 9.8 MB 00:02 2024-05-15T12:31:29.321 INFO:teuthology.orchestra.run.smithi117.stdout:(6/32): python3-ceph-argparse-19.0.0-1882.gae24 388 kB/s | 45 kB 00:00 2024-05-15T12:31:29.430 INFO:teuthology.orchestra.run.smithi117.stdout:(7/32): python3-ceph-common-19.0.0-1882.gae2481 1.2 MB/s | 131 kB 00:00 2024-05-15T12:31:29.530 INFO:teuthology.orchestra.run.smithi117.stdout:(8/32): python3-cephfs-19.0.0-1882.gae248125.el 1.6 MB/s | 162 kB 00:00 2024-05-15T12:31:29.722 INFO:teuthology.orchestra.run.smithi117.stdout:(9/32): python3-rados-19.0.0-1882.gae248125.el9 1.6 MB/s | 321 kB 00:00 2024-05-15T12:31:29.930 INFO:teuthology.orchestra.run.smithi117.stdout:(10/32): python3-rbd-19.0.0-1882.gae248125.el9. 1.4 MB/s | 298 kB 00:00 2024-05-15T12:31:30.056 INFO:teuthology.orchestra.run.smithi117.stdout:(11/32): python3-rgw-19.0.0-1882.gae248125.el9. 795 kB/s | 99 kB 00:00 2024-05-15T12:31:30.198 INFO:teuthology.orchestra.run.smithi117.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 755 kB/s | 107 kB 00:00 2024-05-15T12:31:30.282 INFO:teuthology.orchestra.run.smithi117.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.9 MB/s | 160 kB 00:00 2024-05-15T12:31:30.423 INFO:teuthology.orchestra.run.smithi117.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 319 kB/s | 45 kB 00:00 2024-05-15T12:31:30.632 INFO:teuthology.orchestra.run.smithi117.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 3.1 MB/s | 662 kB 00:00 2024-05-15T12:31:30.716 INFO:teuthology.orchestra.run.smithi117.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.4 MB/s | 292 kB 00:00 2024-05-15T12:31:30.758 INFO:teuthology.orchestra.run.smithi117.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.0 MB/s | 42 kB 00:00 2024-05-15T12:31:30.841 INFO:teuthology.orchestra.run.smithi117.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.6 MB/s | 221 kB 00:00 2024-05-15T12:31:30.883 INFO:teuthology.orchestra.run.smithi117.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 860 kB/s | 36 kB 00:00 2024-05-15T12:31:30.925 INFO:teuthology.orchestra.run.smithi117.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2024-05-15T12:31:31.192 INFO:teuthology.orchestra.run.smithi117.stdout:(21/32): librgw2-19.0.0-1882.gae248125.el9.x86_ 2.5 MB/s | 5.1 MB 00:02 2024-05-15T12:31:31.459 INFO:teuthology.orchestra.run.smithi117.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 579 kB/s | 309 kB 00:00 2024-05-15T12:31:31.573 INFO:teuthology.orchestra.run.smithi178.stdout:(1/32): ceph-base-19.0.0-1882.gae248125.el9.x86 1.6 MB/s | 5.4 MB 00:03 2024-05-15T12:31:31.665 INFO:teuthology.orchestra.run.smithi178.stdout:(2/32): ceph-selinux-19.0.0-1882.gae248125.el9. 275 kB/s | 25 kB 00:00 2024-05-15T12:31:31.710 INFO:teuthology.orchestra.run.smithi117.stdout:(23/32): ceph-common-19.0.0-1882.gae248125.el9. 3.7 MB/s | 20 MB 00:05 2024-05-15T12:31:31.726 INFO:teuthology.orchestra.run.smithi117.stdout:(24/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 97 kB/s | 26 kB 00:00 2024-05-15T12:31:31.827 INFO:teuthology.orchestra.run.smithi117.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 675 kB/s | 67 kB 00:00 2024-05-15T12:31:31.949 INFO:teuthology.orchestra.run.smithi178.stdout:(3/32): libcephfs2-19.0.0-1882.gae248125.el9.x8 2.6 MB/s | 744 kB 00:00 2024-05-15T12:31:32.010 INFO:teuthology.orchestra.run.smithi117.stdout:(26/32): liboath-2.6.7-2.el9.x86_64.rpm 162 kB/s | 49 kB 00:00 2024-05-15T12:31:32.060 INFO:teuthology.orchestra.run.smithi117.stdout:(27/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 3.5 MB/s | 839 kB 00:00 2024-05-15T12:31:32.141 INFO:teuthology.orchestra.run.smithi178.stdout:(4/32): libradosstriper1-19.0.0-1882.gae248125. 2.6 MB/s | 501 kB 00:00 2024-05-15T12:31:32.202 INFO:teuthology.orchestra.run.smithi117.stdout:(28/32): libarrow-9.0.0-11.el9.x86_64.rpm 4.4 MB/s | 4.4 MB 00:01 2024-05-15T12:31:32.261 INFO:teuthology.orchestra.run.smithi117.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 7.9 MB/s | 1.6 MB 00:00 2024-05-15T12:31:32.294 INFO:teuthology.orchestra.run.smithi117.stdout:(30/32): re2-20211101-3.el9.x86_64.rpm 674 kB/s | 192 kB 00:00 2024-05-15T12:31:33.516 INFO:teuthology.orchestra.run.smithi178.stdout:(5/32): librgw2-19.0.0-1882.gae248125.el9.x86_6 3.7 MB/s | 5.1 MB 00:01 2024-05-15T12:31:33.616 INFO:teuthology.orchestra.run.smithi178.stdout:(6/32): python3-ceph-argparse-19.0.0-1882.gae24 452 kB/s | 45 kB 00:00 2024-05-15T12:31:33.733 INFO:teuthology.orchestra.run.smithi178.stdout:(7/32): python3-ceph-common-19.0.0-1882.gae2481 1.1 MB/s | 131 kB 00:00 2024-05-15T12:31:33.850 INFO:teuthology.orchestra.run.smithi178.stdout:(8/32): python3-cephfs-19.0.0-1882.gae248125.el 1.4 MB/s | 162 kB 00:00 2024-05-15T12:31:33.975 INFO:teuthology.orchestra.run.smithi178.stdout:(9/32): python3-rados-19.0.0-1882.gae248125.el9 2.5 MB/s | 321 kB 00:00 2024-05-15T12:31:34.044 INFO:teuthology.orchestra.run.smithi117.stdout:(31/32): librados2-19.0.0-1882.gae248125.el9.x8 1.8 MB/s | 3.4 MB 00:01 2024-05-15T12:31:34.092 INFO:teuthology.orchestra.run.smithi178.stdout:(10/32): python3-rbd-19.0.0-1882.gae248125.el9. 2.5 MB/s | 298 kB 00:00 2024-05-15T12:31:34.193 INFO:teuthology.orchestra.run.smithi178.stdout:(11/32): python3-rgw-19.0.0-1882.gae248125.el9. 995 kB/s | 99 kB 00:00 2024-05-15T12:31:34.310 INFO:teuthology.orchestra.run.smithi178.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 918 kB/s | 107 kB 00:00 2024-05-15T12:31:34.368 INFO:teuthology.orchestra.run.smithi178.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 160 kB 00:00 2024-05-15T12:31:34.410 INFO:teuthology.orchestra.run.smithi178.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-05-15T12:31:34.519 INFO:teuthology.orchestra.run.smithi178.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 6.0 MB/s | 662 kB 00:00 2024-05-15T12:31:34.561 INFO:teuthology.orchestra.run.smithi178.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.9 MB/s | 292 kB 00:00 2024-05-15T12:31:34.603 INFO:teuthology.orchestra.run.smithi178.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.0 MB/s | 42 kB 00:00 2024-05-15T12:31:34.644 INFO:teuthology.orchestra.run.smithi178.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 5.2 MB/s | 221 kB 00:00 2024-05-15T12:31:34.686 INFO:teuthology.orchestra.run.smithi178.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 861 kB/s | 36 kB 00:00 2024-05-15T12:31:34.728 INFO:teuthology.orchestra.run.smithi178.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2024-05-15T12:31:34.945 INFO:teuthology.orchestra.run.smithi178.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.4 MB/s | 309 kB 00:00 2024-05-15T12:31:35.146 INFO:teuthology.orchestra.run.smithi178.stdout:(22/32): ceph-common-19.0.0-1882.gae248125.el9. 2.9 MB/s | 20 MB 00:06 2024-05-15T12:31:35.211 INFO:teuthology.orchestra.run.smithi117.stdout:(32/32): librbd1-19.0.0-1882.gae248125.el9.x86_ 1.1 MB/s | 3.1 MB 00:02 2024-05-15T12:31:35.214 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:31:35.214 INFO:teuthology.orchestra.run.smithi117.stdout:Total 6.3 MB/s | 58 MB 00:09 2024-05-15T12:31:35.296 INFO:teuthology.orchestra.run.smithi178.stdout:(23/32): libarrow-9.0.0-11.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-05-15T12:31:35.320 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-05-15T12:31:35.321 INFO:teuthology.orchestra.run.smithi178.stdout:(24/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 150 kB/s | 26 kB 00:00 2024-05-15T12:31:35.346 INFO:teuthology.orchestra.run.smithi178.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 971 kB/s | 49 kB 00:00 2024-05-15T12:31:35.357 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-05-15T12:31:35.357 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-05-15T12:31:35.371 INFO:teuthology.orchestra.run.smithi178.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-05-15T12:31:35.405 INFO:teuthology.orchestra.run.smithi178.stdout:(27/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 14 MB/s | 839 kB 00:00 2024-05-15T12:31:35.447 INFO:teuthology.orchestra.run.smithi178.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-05-15T12:31:35.489 INFO:teuthology.orchestra.run.smithi178.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2024-05-15T12:31:35.973 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-05-15T12:31:35.974 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-05-15T12:31:36.290 INFO:teuthology.orchestra.run.smithi178.stdout:(30/32): librbd1-19.0.0-1882.gae248125.el9.x86_ 3.9 MB/s | 3.1 MB 00:00 2024-05-15T12:31:36.406 INFO:teuthology.orchestra.run.smithi178.stdout:(31/32): librados2-19.0.0-1882.gae248125.el9.x8 3.5 MB/s | 3.4 MB 00:00 2024-05-15T12:31:37.393 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-05-15T12:31:37.470 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-05-15T12:31:37.499 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-05-15T12:31:37.533 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-05-15T12:31:37.556 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-05-15T12:31:37.581 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-05-15T12:31:37.741 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-05-15T12:31:37.795 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-05-15T12:31:38.138 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-05-15T12:31:38.259 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : librados2-2:19.0.0-1882.gae248125.el9.x86_64 7/34 2024-05-15T12:31:38.352 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librados2-2:19.0.0-1882.gae248125.el9.x86_64 7/34 2024-05-15T12:31:38.442 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-rados-2:19.0.0-1882.gae248125.el9.x86_64 8/34 2024-05-15T12:31:38.469 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libcephfs2-2:19.0.0-1882.gae248125.el9.x86_64 9/34 2024-05-15T12:31:38.524 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: libcephfs2-2:19.0.0-1882.gae248125.el9.x86_64 9/34 2024-05-15T12:31:38.552 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-05-15T12:31:38.596 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-ceph-argparse-2:19.0.0-1882.gae248125.el9. 11/34 2024-05-15T12:31:38.666 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cephfs-2:19.0.0-1882.gae248125.el9.x86_64 12/34 2024-05-15T12:31:38.688 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libradosstriper1-2:19.0.0-1882.gae248125.el9.x86_6 13/34 2024-05-15T12:31:38.752 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1882.gae248125.el9.x86_6 13/34 2024-05-15T12:31:38.787 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-05-15T12:31:38.898 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-05-15T12:31:38.946 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-05-15T12:31:39.005 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-05-15T12:31:39.044 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-05-15T12:31:39.073 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-05-15T12:31:39.106 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-05-15T12:31:39.436 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-05-15T12:31:39.486 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : librbd1-2:19.0.0-1882.gae248125.el9.x86_64 22/34 2024-05-15T12:31:39.576 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librbd1-2:19.0.0-1882.gae248125.el9.x86_64 22/34 2024-05-15T12:31:39.607 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-rbd-2:19.0.0-1882.gae248125.el9.x86_64 23/34 2024-05-15T12:31:39.643 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-05-15T12:31:40.098 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-05-15T12:31:40.129 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librgw2-2:19.0.0-1882.gae248125.el9.x86_64 26/34 2024-05-15T12:31:40.201 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librgw2-2:19.0.0-1882.gae248125.el9.x86_64 26/34 2024-05-15T12:31:40.270 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-rgw-2:19.0.0-1882.gae248125.el9.x86_64 27/34 2024-05-15T12:31:40.299 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-ceph-common-2:19.0.0-1882.gae248125.el9.x8 28/34 2024-05-15T12:31:41.906 INFO:teuthology.orchestra.run.smithi178.stdout:(32/32): ceph-radosgw-19.0.0-1882.gae248125.el9 729 kB/s | 9.8 MB 00:13 2024-05-15T12:31:41.911 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:31:41.911 INFO:teuthology.orchestra.run.smithi178.stdout:Total 4.2 MB/s | 58 MB 00:13 2024-05-15T12:31:42.015 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-05-15T12:31:42.051 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-05-15T12:31:42.052 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-05-15T12:31:42.485 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-common-2:19.0.0-1882.gae248125.el9.x86_64 29/34 2024-05-15T12:31:42.520 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-common-2:19.0.0-1882.gae248125.el9.x86_64 29/34 2024-05-15T12:31:42.651 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-05-15T12:31:42.651 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-05-15T12:31:43.070 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-common-2:19.0.0-1882.gae248125.el9.x86_64 29/34 2024-05-15T12:31:43.108 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-base-2:19.0.0-1882.gae248125.el9.x86_64 30/34 2024-05-15T12:31:43.204 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-base-2:19.0.0-1882.gae248125.el9.x86_64 30/34 2024-05-15T12:31:43.204 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-15T12:31:43.204 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-15T12:31:43.204 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:31:43.234 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-selinux-2:19.0.0-1882.gae248125.el9.x86_64 31/34 2024-05-15T12:31:44.082 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-05-15T12:31:44.160 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-05-15T12:31:44.194 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-05-15T12:31:44.223 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-05-15T12:31:44.250 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-05-15T12:31:44.273 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-05-15T12:31:44.428 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-05-15T12:31:44.485 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-05-15T12:31:44.859 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-05-15T12:31:44.913 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : librados2-2:19.0.0-1882.gae248125.el9.x86_64 7/34 2024-05-15T12:31:45.022 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librados2-2:19.0.0-1882.gae248125.el9.x86_64 7/34 2024-05-15T12:31:45.112 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rados-2:19.0.0-1882.gae248125.el9.x86_64 8/34 2024-05-15T12:31:45.139 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libcephfs2-2:19.0.0-1882.gae248125.el9.x86_64 9/34 2024-05-15T12:31:45.202 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libcephfs2-2:19.0.0-1882.gae248125.el9.x86_64 9/34 2024-05-15T12:31:45.240 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-05-15T12:31:45.287 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-ceph-argparse-2:19.0.0-1882.gae248125.el9. 11/34 2024-05-15T12:31:45.352 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cephfs-2:19.0.0-1882.gae248125.el9.x86_64 12/34 2024-05-15T12:31:45.375 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libradosstriper1-2:19.0.0-1882.gae248125.el9.x86_6 13/34 2024-05-15T12:31:45.439 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1882.gae248125.el9.x86_6 13/34 2024-05-15T12:31:45.473 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-05-15T12:31:45.583 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-05-15T12:31:45.624 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-05-15T12:31:45.665 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-05-15T12:31:45.697 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-05-15T12:31:45.726 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-05-15T12:31:45.759 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-05-15T12:31:46.086 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-05-15T12:31:46.130 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : librbd1-2:19.0.0-1882.gae248125.el9.x86_64 22/34 2024-05-15T12:31:46.222 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librbd1-2:19.0.0-1882.gae248125.el9.x86_64 22/34 2024-05-15T12:31:46.253 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rbd-2:19.0.0-1882.gae248125.el9.x86_64 23/34 2024-05-15T12:31:46.297 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-05-15T12:31:46.772 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-05-15T12:31:46.800 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librgw2-2:19.0.0-1882.gae248125.el9.x86_64 26/34 2024-05-15T12:31:46.863 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librgw2-2:19.0.0-1882.gae248125.el9.x86_64 26/34 2024-05-15T12:31:46.939 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rgw-2:19.0.0-1882.gae248125.el9.x86_64 27/34 2024-05-15T12:31:46.969 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-ceph-common-2:19.0.0-1882.gae248125.el9.x8 28/34 2024-05-15T12:31:49.128 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-common-2:19.0.0-1882.gae248125.el9.x86_64 29/34 2024-05-15T12:31:49.166 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-common-2:19.0.0-1882.gae248125.el9.x86_64 29/34 2024-05-15T12:31:49.715 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-common-2:19.0.0-1882.gae248125.el9.x86_64 29/34 2024-05-15T12:31:49.753 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-base-2:19.0.0-1882.gae248125.el9.x86_64 30/34 2024-05-15T12:31:49.860 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-base-2:19.0.0-1882.gae248125.el9.x86_64 30/34 2024-05-15T12:31:49.860 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-15T12:31:49.860 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-15T12:31:49.860 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:31:49.888 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-selinux-2:19.0.0-1882.gae248125.el9.x86_64 31/34 2024-05-15T12:31:52.720 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1882.gae248125.el9.x86_64 31/34 2024-05-15T12:31:52.720 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /sys 2024-05-15T12:31:52.720 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /proc 2024-05-15T12:31:52.720 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /mnt 2024-05-15T12:31:52.720 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /var/tmp 2024-05-15T12:31:52.720 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /home 2024-05-15T12:31:52.720 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /root 2024-05-15T12:31:52.720 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /tmp 2024-05-15T12:31:52.720 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:31:53.643 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-radosgw-2:19.0.0-1882.gae248125.el9.x86_64 32/34 2024-05-15T12:31:53.668 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1882.gae248125.el9.x86_64 32/34 2024-05-15T12:31:53.668 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-15T12:31:53.668 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-15T12:31:53.668 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-15T12:31:53.669 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-15T12:31:53.669 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:31:53.670 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-05-15T12:31:53.753 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-05-15T12:31:53.754 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-05-15T12:31:54.891 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-05-15T12:31:54.892 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-base-2:19.0.0-1882.gae248125.el9.x86_64 1/34 2024-05-15T12:31:54.892 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-common-2:19.0.0-1882.gae248125.el9.x86_64 2/34 2024-05-15T12:31:54.892 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-radosgw-2:19.0.0-1882.gae248125.el9.x86_64 3/34 2024-05-15T12:31:54.892 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-selinux-2:19.0.0-1882.gae248125.el9.x86_64 4/34 2024-05-15T12:31:54.892 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libcephfs2-2:19.0.0-1882.gae248125.el9.x86_64 5/34 2024-05-15T12:31:54.892 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libradosstriper1-2:19.0.0-1882.gae248125.el9.x86_6 6/34 2024-05-15T12:31:54.892 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librgw2-2:19.0.0-1882.gae248125.el9.x86_64 7/34 2024-05-15T12:31:54.892 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1882.gae248125.el9. 8/34 2024-05-15T12:31:54.892 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-ceph-common-2:19.0.0-1882.gae248125.el9.x8 9/34 2024-05-15T12:31:54.892 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cephfs-2:19.0.0-1882.gae248125.el9.x86_64 10/34 2024-05-15T12:31:54.892 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-rados-2:19.0.0-1882.gae248125.el9.x86_64 11/34 2024-05-15T12:31:54.892 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-rbd-2:19.0.0-1882.gae248125.el9.x86_64 12/34 2024-05-15T12:31:54.893 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-rgw-2:19.0.0-1882.gae248125.el9.x86_64 13/34 2024-05-15T12:31:54.893 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-05-15T12:31:54.893 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-05-15T12:31:54.893 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-05-15T12:31:54.893 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-05-15T12:31:54.893 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-05-15T12:31:54.893 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-05-15T12:31:54.893 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-05-15T12:31:54.893 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-05-15T12:31:54.893 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-05-15T12:31:54.893 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-05-15T12:31:54.893 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-05-15T12:31:54.893 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-05-15T12:31:54.893 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-05-15T12:31:54.894 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-05-15T12:31:54.895 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-05-15T12:31:54.895 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-05-15T12:31:54.895 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-05-15T12:31:54.895 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librados2-2:19.0.0-1882.gae248125.el9.x86_64 31/34 2024-05-15T12:31:54.895 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-05-15T12:31:54.895 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librbd1-2:19.0.0-1882.gae248125.el9.x86_64 33/34 2024-05-15T12:31:55.795 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-05-15T12:31:55.795 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:31:55.795 INFO:teuthology.orchestra.run.smithi117.stdout:Upgraded: 2024-05-15T12:31:55.795 INFO:teuthology.orchestra.run.smithi117.stdout: librados2-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:31:55.795 INFO:teuthology.orchestra.run.smithi117.stdout: librbd1-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:31:55.795 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-05-15T12:31:55.795 INFO:teuthology.orchestra.run.smithi117.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-15T12:31:55.795 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-base-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:31:55.795 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-common-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:31:55.795 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-radosgw-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:31:55.796 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-selinux-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:31:55.796 INFO:teuthology.orchestra.run.smithi117.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-15T12:31:55.796 INFO:teuthology.orchestra.run.smithi117.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-15T12:31:55.796 INFO:teuthology.orchestra.run.smithi117.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-15T12:31:55.796 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs2-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:31:55.796 INFO:teuthology.orchestra.run.smithi117.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-15T12:31:55.796 INFO:teuthology.orchestra.run.smithi117.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-15T12:31:55.796 INFO:teuthology.orchestra.run.smithi117.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-15T12:31:55.796 INFO:teuthology.orchestra.run.smithi117.stdout: libradosstriper1-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:31:55.796 INFO:teuthology.orchestra.run.smithi117.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-15T12:31:55.797 INFO:teuthology.orchestra.run.smithi117.stdout: librgw2-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:31:55.797 INFO:teuthology.orchestra.run.smithi117.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-15T12:31:55.797 INFO:teuthology.orchestra.run.smithi117.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-15T12:31:55.797 INFO:teuthology.orchestra.run.smithi117.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-15T12:31:55.797 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ceph-argparse-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:31:55.797 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ceph-common-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:31:55.797 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cephfs-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:31:55.797 INFO:teuthology.orchestra.run.smithi117.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-15T12:31:55.797 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rados-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:31:55.797 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rbd-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:31:55.797 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rgw-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:31:55.797 INFO:teuthology.orchestra.run.smithi117.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-15T12:31:55.797 INFO:teuthology.orchestra.run.smithi117.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-15T12:31:55.798 INFO:teuthology.orchestra.run.smithi117.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-15T12:31:55.798 INFO:teuthology.orchestra.run.smithi117.stdout: re2-1:20211101-3.el9.x86_64 2024-05-15T12:31:55.798 INFO:teuthology.orchestra.run.smithi117.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-15T12:31:55.798 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:31:55.798 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-05-15T12:31:56.149 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-test 2024-05-15T12:31:56.729 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:00:33 ago on Wed 15 May 2024 12:31:23 PM UTC. 2024-05-15T12:31:56.851 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-05-15T12:31:56.853 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:31:56.853 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-05-15T12:31:56.853 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:31:56.853 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-05-15T12:31:56.853 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-test x86_64 2:19.0.0-1882.gae248125.el9 ceph 47 M 2024-05-15T12:31:56.853 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-05-15T12:31:56.853 INFO:teuthology.orchestra.run.smithi117.stdout: libcephsqlite x86_64 2:19.0.0-1882.gae248125.el9 ceph 163 k 2024-05-15T12:31:56.853 INFO:teuthology.orchestra.run.smithi117.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-15T12:31:56.853 INFO:teuthology.orchestra.run.smithi117.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-15T12:31:56.853 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:31:56.853 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-05-15T12:31:56.853 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:31:56.853 INFO:teuthology.orchestra.run.smithi117.stdout:Install 4 Packages 2024-05-15T12:31:56.853 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:31:56.853 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 48 M 2024-05-15T12:31:56.853 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 204 M 2024-05-15T12:31:56.854 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-05-15T12:31:57.200 INFO:teuthology.orchestra.run.smithi117.stdout:(1/4): libcephsqlite-19.0.0-1882.gae248125.el9. 707 kB/s | 163 kB 00:00 2024-05-15T12:31:57.800 INFO:teuthology.orchestra.run.smithi117.stdout:(2/4): socat-1.7.4.1-5.el9.x86_64.rpm 367 kB/s | 305 kB 00:00 2024-05-15T12:31:57.892 INFO:teuthology.orchestra.run.smithi117.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 95 kB/s | 64 kB 00:00 2024-05-15T12:31:59.150 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1882.gae248125.el9.x86_64 31/34 2024-05-15T12:31:59.150 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /sys 2024-05-15T12:31:59.150 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /proc 2024-05-15T12:31:59.150 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /mnt 2024-05-15T12:31:59.150 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /var/tmp 2024-05-15T12:31:59.150 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /home 2024-05-15T12:31:59.150 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /root 2024-05-15T12:31:59.150 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /tmp 2024-05-15T12:31:59.150 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:31:59.601 INFO:teuthology.orchestra.run.smithi117.stdout:(4/4): ceph-test-19.0.0-1882.gae248125.el9.x86_ 18 MB/s | 47 MB 00:02 2024-05-15T12:31:59.603 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:31:59.603 INFO:teuthology.orchestra.run.smithi117.stdout:Total 17 MB/s | 48 MB 00:02 2024-05-15T12:31:59.631 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-05-15T12:31:59.651 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-05-15T12:31:59.651 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-05-15T12:32:00.012 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-05-15T12:32:00.012 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-05-15T12:32:00.065 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-radosgw-2:19.0.0-1882.gae248125.el9.x86_64 32/34 2024-05-15T12:32:00.091 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1882.gae248125.el9.x86_64 32/34 2024-05-15T12:32:00.091 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-15T12:32:00.091 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-15T12:32:00.091 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-15T12:32:00.091 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-15T12:32:00.091 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:32:00.093 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-05-15T12:32:00.159 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-05-15T12:32:00.159 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-05-15T12:32:00.534 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-05-15T12:32:00.618 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-05-15T12:32:00.666 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-05-15T12:32:00.694 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libcephsqlite-2:19.0.0-1882.gae248125.el9.x86_64 3/4 2024-05-15T12:32:01.305 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-05-15T12:32:01.305 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-base-2:19.0.0-1882.gae248125.el9.x86_64 1/34 2024-05-15T12:32:01.305 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-common-2:19.0.0-1882.gae248125.el9.x86_64 2/34 2024-05-15T12:32:01.305 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-radosgw-2:19.0.0-1882.gae248125.el9.x86_64 3/34 2024-05-15T12:32:01.305 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-selinux-2:19.0.0-1882.gae248125.el9.x86_64 4/34 2024-05-15T12:32:01.305 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libcephfs2-2:19.0.0-1882.gae248125.el9.x86_64 5/34 2024-05-15T12:32:01.305 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libradosstriper1-2:19.0.0-1882.gae248125.el9.x86_6 6/34 2024-05-15T12:32:01.305 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librgw2-2:19.0.0-1882.gae248125.el9.x86_64 7/34 2024-05-15T12:32:01.305 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1882.gae248125.el9. 8/34 2024-05-15T12:32:01.305 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-ceph-common-2:19.0.0-1882.gae248125.el9.x8 9/34 2024-05-15T12:32:01.305 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cephfs-2:19.0.0-1882.gae248125.el9.x86_64 10/34 2024-05-15T12:32:01.306 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rados-2:19.0.0-1882.gae248125.el9.x86_64 11/34 2024-05-15T12:32:01.306 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rbd-2:19.0.0-1882.gae248125.el9.x86_64 12/34 2024-05-15T12:32:01.306 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rgw-2:19.0.0-1882.gae248125.el9.x86_64 13/34 2024-05-15T12:32:01.306 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-05-15T12:32:01.306 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-05-15T12:32:01.306 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-05-15T12:32:01.306 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-05-15T12:32:01.306 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-05-15T12:32:01.306 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-05-15T12:32:01.306 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-05-15T12:32:01.306 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-05-15T12:32:01.306 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-05-15T12:32:01.306 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-05-15T12:32:01.306 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-05-15T12:32:01.306 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-05-15T12:32:01.306 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-05-15T12:32:01.307 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-05-15T12:32:01.307 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-05-15T12:32:01.307 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-05-15T12:32:01.307 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-05-15T12:32:01.308 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librados2-2:19.0.0-1882.gae248125.el9.x86_64 31/34 2024-05-15T12:32:01.308 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-05-15T12:32:01.308 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librbd1-2:19.0.0-1882.gae248125.el9.x86_64 33/34 2024-05-15T12:32:02.121 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-05-15T12:32:02.121 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:32:02.121 INFO:teuthology.orchestra.run.smithi178.stdout:Upgraded: 2024-05-15T12:32:02.121 INFO:teuthology.orchestra.run.smithi178.stdout: librados2-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:32:02.121 INFO:teuthology.orchestra.run.smithi178.stdout: librbd1-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:32:02.121 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-05-15T12:32:02.122 INFO:teuthology.orchestra.run.smithi178.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-15T12:32:02.122 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-base-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:32:02.122 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-common-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:32:02.122 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-radosgw-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:32:02.122 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-selinux-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:32:02.122 INFO:teuthology.orchestra.run.smithi178.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-15T12:32:02.122 INFO:teuthology.orchestra.run.smithi178.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-15T12:32:02.122 INFO:teuthology.orchestra.run.smithi178.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-15T12:32:02.122 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs2-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:32:02.122 INFO:teuthology.orchestra.run.smithi178.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-15T12:32:02.122 INFO:teuthology.orchestra.run.smithi178.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-15T12:32:02.122 INFO:teuthology.orchestra.run.smithi178.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-15T12:32:02.122 INFO:teuthology.orchestra.run.smithi178.stdout: libradosstriper1-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:32:02.122 INFO:teuthology.orchestra.run.smithi178.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-15T12:32:02.122 INFO:teuthology.orchestra.run.smithi178.stdout: librgw2-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:32:02.123 INFO:teuthology.orchestra.run.smithi178.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-15T12:32:02.123 INFO:teuthology.orchestra.run.smithi178.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-15T12:32:02.123 INFO:teuthology.orchestra.run.smithi178.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-15T12:32:02.123 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-argparse-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:32:02.123 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-common-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:32:02.123 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cephfs-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:32:02.123 INFO:teuthology.orchestra.run.smithi178.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-15T12:32:02.123 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rados-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:32:02.123 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rbd-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:32:02.123 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rgw-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:32:02.123 INFO:teuthology.orchestra.run.smithi178.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-15T12:32:02.123 INFO:teuthology.orchestra.run.smithi178.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-15T12:32:02.123 INFO:teuthology.orchestra.run.smithi178.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-15T12:32:02.123 INFO:teuthology.orchestra.run.smithi178.stdout: re2-1:20211101-3.el9.x86_64 2024-05-15T12:32:02.123 INFO:teuthology.orchestra.run.smithi178.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-15T12:32:02.123 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:32:02.123 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-05-15T12:32:02.482 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-test 2024-05-15T12:32:03.060 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:00:38 ago on Wed 15 May 2024 12:31:25 PM UTC. 2024-05-15T12:32:03.183 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-05-15T12:32:03.184 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:32:03.184 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-05-15T12:32:03.184 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:32:03.184 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-05-15T12:32:03.184 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-test x86_64 2:19.0.0-1882.gae248125.el9 ceph 47 M 2024-05-15T12:32:03.184 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-05-15T12:32:03.185 INFO:teuthology.orchestra.run.smithi178.stdout: libcephsqlite x86_64 2:19.0.0-1882.gae248125.el9 ceph 163 k 2024-05-15T12:32:03.185 INFO:teuthology.orchestra.run.smithi178.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-15T12:32:03.185 INFO:teuthology.orchestra.run.smithi178.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-15T12:32:03.185 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:32:03.185 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-05-15T12:32:03.185 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:32:03.185 INFO:teuthology.orchestra.run.smithi178.stdout:Install 4 Packages 2024-05-15T12:32:03.185 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:32:03.185 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 48 M 2024-05-15T12:32:03.185 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 204 M 2024-05-15T12:32:03.185 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-05-15T12:32:03.587 INFO:teuthology.orchestra.run.smithi178.stdout:(1/4): socat-1.7.4.1-5.el9.x86_64.rpm 1.1 MB/s | 305 kB 00:00 2024-05-15T12:32:03.646 INFO:teuthology.orchestra.run.smithi178.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-05-15T12:32:03.680 INFO:teuthology.orchestra.run.smithi178.stdout:(3/4): libcephsqlite-19.0.0-1882.gae248125.el9. 440 kB/s | 163 kB 00:00 2024-05-15T12:32:05.162 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1882.gae248125.el9.x86_64 3/4 2024-05-15T12:32:05.250 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-test-2:19.0.0-1882.gae248125.el9.x86_64 4/4 2024-05-15T12:32:05.614 INFO:teuthology.orchestra.run.smithi178.stdout:(4/4): ceph-test-19.0.0-1882.gae248125.el9.x86_ 21 MB/s | 47 MB 00:02 2024-05-15T12:32:05.616 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:32:05.616 INFO:teuthology.orchestra.run.smithi178.stdout:Total 20 MB/s | 48 MB 00:02 2024-05-15T12:32:05.644 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-05-15T12:32:05.664 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-05-15T12:32:05.664 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-05-15T12:32:06.024 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-05-15T12:32:06.024 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-05-15T12:32:06.546 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-05-15T12:32:06.640 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-05-15T12:32:06.679 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-05-15T12:32:06.707 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libcephsqlite-2:19.0.0-1882.gae248125.el9.x86_64 3/4 2024-05-15T12:32:07.687 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-test-2:19.0.0-1882.gae248125.el9.x86_64 4/4 2024-05-15T12:32:07.687 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-test-2:19.0.0-1882.gae248125.el9.x86_64 1/4 2024-05-15T12:32:07.687 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libcephsqlite-2:19.0.0-1882.gae248125.el9.x86_64 2/4 2024-05-15T12:32:07.687 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-05-15T12:32:07.951 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-05-15T12:32:07.952 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:32:07.952 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-05-15T12:32:07.952 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-test-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:32:07.952 INFO:teuthology.orchestra.run.smithi117.stdout: libcephsqlite-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:32:07.952 INFO:teuthology.orchestra.run.smithi117.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-15T12:32:07.952 INFO:teuthology.orchestra.run.smithi117.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-15T12:32:07.952 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:32:07.952 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-05-15T12:32:08.218 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph 2024-05-15T12:32:08.799 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:00:45 ago on Wed 15 May 2024 12:31:23 PM UTC. 2024-05-15T12:32:08.931 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-05-15T12:32:08.934 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:32:08.934 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-05-15T12:32:08.934 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:32:08.934 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-05-15T12:32:08.935 INFO:teuthology.orchestra.run.smithi117.stdout: ceph x86_64 2:19.0.0-1882.gae248125.el9 ceph 6.6 k 2024-05-15T12:32:08.935 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-05-15T12:32:08.935 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mds x86_64 2:19.0.0-1882.gae248125.el9 ceph 2.1 M 2024-05-15T12:32:08.935 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr x86_64 2:19.0.0-1882.gae248125.el9 ceph 1.5 M 2024-05-15T12:32:08.935 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1882.gae248125.el9 ceph-noarch 248 k 2024-05-15T12:32:08.935 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mon x86_64 2:19.0.0-1882.gae248125.el9 ceph 4.7 M 2024-05-15T12:32:08.935 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-osd x86_64 2:19.0.0-1882.gae248125.el9 ceph 17 M 2024-05-15T12:32:08.935 INFO:teuthology.orchestra.run.smithi117.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-15T12:32:08.935 INFO:teuthology.orchestra.run.smithi117.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-15T12:32:08.935 INFO:teuthology.orchestra.run.smithi117.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-05-15T12:32:08.935 INFO:teuthology.orchestra.run.smithi117.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-15T12:32:08.935 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-15T12:32:08.935 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-15T12:32:08.935 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-15T12:32:08.935 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-15T12:32:08.936 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-15T12:32:08.936 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-15T12:32:08.936 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-collections 2024-05-15T12:32:08.936 INFO:teuthology.orchestra.run.smithi117.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-05-15T12:32:08.936 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-15T12:32:08.936 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-15T12:32:08.936 INFO:teuthology.orchestra.run.smithi117.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-15T12:32:08.936 INFO:teuthology.orchestra.run.smithi117.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-15T12:32:08.936 INFO:teuthology.orchestra.run.smithi117.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-15T12:32:08.936 INFO:teuthology.orchestra.run.smithi117.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-15T12:32:08.936 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-15T12:32:08.936 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-15T12:32:08.936 INFO:teuthology.orchestra.run.smithi117.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-15T12:32:08.936 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-15T12:32:08.937 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-15T12:32:08.937 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-15T12:32:08.937 INFO:teuthology.orchestra.run.smithi117.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-15T12:32:08.937 INFO:teuthology.orchestra.run.smithi117.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-15T12:32:08.937 INFO:teuthology.orchestra.run.smithi117.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-15T12:32:08.937 INFO:teuthology.orchestra.run.smithi117.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-15T12:32:08.937 INFO:teuthology.orchestra.run.smithi117.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-15T12:32:08.937 INFO:teuthology.orchestra.run.smithi117.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-15T12:32:08.937 INFO:teuthology.orchestra.run.smithi117.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-15T12:32:08.937 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:32:08.937 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-05-15T12:32:08.937 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:32:08.937 INFO:teuthology.orchestra.run.smithi117.stdout:Install 36 Packages 2024-05-15T12:32:08.937 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:32:08.938 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 30 M 2024-05-15T12:32:08.938 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 107 M 2024-05-15T12:32:08.938 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-05-15T12:32:09.693 INFO:teuthology.orchestra.run.smithi117.stdout:(1/36): ceph-19.0.0-1882.gae248125.el9.x86_64.r 30 kB/s | 6.6 kB 00:00 2024-05-15T12:32:09.994 INFO:teuthology.orchestra.run.smithi117.stdout:(2/36): ceph-mds-19.0.0-1882.gae248125.el9.x86_ 4.1 MB/s | 2.1 MB 00:00 2024-05-15T12:32:10.027 INFO:teuthology.orchestra.run.smithi117.stdout:(3/36): ceph-mgr-19.0.0-1882.gae248125.el9.x86_ 2.7 MB/s | 1.5 MB 00:00 2024-05-15T12:32:10.136 INFO:teuthology.orchestra.run.smithi117.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1882.gae24 2.2 MB/s | 248 kB 00:00 2024-05-15T12:32:10.303 INFO:teuthology.orchestra.run.smithi117.stdout:(5/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.5 MB/s | 253 kB 00:00 2024-05-15T12:32:10.370 INFO:teuthology.orchestra.run.smithi117.stdout:(6/36): ceph-mon-19.0.0-1882.gae248125.el9.x86_ 6.9 MB/s | 4.7 MB 00:00 2024-05-15T12:32:10.436 INFO:teuthology.orchestra.run.smithi117.stdout:(7/36): python3-cryptography-36.0.1-4.el9.x86_6 9.2 MB/s | 1.2 MB 00:00 2024-05-15T12:32:10.487 INFO:teuthology.orchestra.run.smithi117.stdout:(8/36): python3-pycparser-2.20-6.el9.noarch.rpm 2.7 MB/s | 135 kB 00:00 2024-05-15T12:32:10.537 INFO:teuthology.orchestra.run.smithi117.stdout:(9/36): python3-ply-3.11-14.el9.noarch.rpm 638 kB/s | 106 kB 00:00 2024-05-15T12:32:10.562 INFO:teuthology.orchestra.run.smithi117.stdout:(10/36): python3-requests-2.25.1-8.el9.noarch.r 1.6 MB/s | 125 kB 00:00 2024-05-15T12:32:10.629 INFO:teuthology.orchestra.run.smithi117.stdout:(11/36): python3-urllib3-1.26.5-5.el9.noarch.rp 2.3 MB/s | 215 kB 00:00 2024-05-15T12:32:10.671 INFO:teuthology.orchestra.run.smithi117.stdout:(12/36): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 188 kB 00:00 2024-05-15T12:32:10.713 INFO:teuthology.orchestra.run.smithi117.stdout:(13/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-05-15T12:32:10.738 INFO:teuthology.orchestra.run.smithi117.stdout:(14/36): python3-markupsafe-1.1.1-12.el9.x86_64 520 kB/s | 35 kB 00:00 2024-05-15T12:32:10.939 INFO:teuthology.orchestra.run.smithi117.stdout:(15/36): ceph-osd-19.0.0-1882.gae248125.el9.x86 18 MB/s | 17 MB 00:00 2024-05-15T12:32:10.963 INFO:teuthology.orchestra.run.smithi117.stdout:(16/36): python3-pytz-2021.1-5.el9.noarch.rpm 204 kB/s | 51 kB 00:00 2024-05-15T12:32:10.989 INFO:teuthology.orchestra.run.smithi117.stdout:(17/36): python3-toml-0.10.2-6.el9.noarch.rpm 166 kB/s | 42 kB 00:00 2024-05-15T12:32:11.064 INFO:teuthology.orchestra.run.smithi117.stdout:(18/36): lua-devel-5.4.4-4.el9.x86_64.rpm 178 kB/s | 22 kB 00:00 2024-05-15T12:32:11.197 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1882.gae248125.el9.x86_64 3/4 2024-05-15T12:32:11.272 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-test-2:19.0.0-1882.gae248125.el9.x86_64 4/4 2024-05-15T12:32:11.315 INFO:teuthology.orchestra.run.smithi117.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 134 kB/s | 43 kB 00:00 2024-05-15T12:32:11.448 INFO:teuthology.orchestra.run.smithi117.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 329 kB/s | 151 kB 00:00 2024-05-15T12:32:11.540 INFO:teuthology.orchestra.run.smithi117.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 361 kB/s | 172 kB 00:00 2024-05-15T12:32:11.565 INFO:teuthology.orchestra.run.smithi117.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 91 kB/s | 11 kB 00:00 2024-05-15T12:32:11.649 INFO:teuthology.orchestra.run.smithi117.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 163 kB/s | 18 kB 00:00 2024-05-15T12:32:11.674 INFO:teuthology.orchestra.run.smithi117.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 213 kB/s | 23 kB 00:00 2024-05-15T12:32:11.699 INFO:teuthology.orchestra.run.smithi117.stdout:(25/36): python3-cherrypy-18.6.1-2.el9.noarch.r 932 kB/s | 358 kB 00:00 2024-05-15T12:32:11.749 INFO:teuthology.orchestra.run.smithi117.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 194 kB/s | 19 kB 00:00 2024-05-15T12:32:11.775 INFO:teuthology.orchestra.run.smithi117.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 195 kB/s | 20 kB 00:00 2024-05-15T12:32:11.850 INFO:teuthology.orchestra.run.smithi117.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 307 kB/s | 46 kB 00:00 2024-05-15T12:32:11.877 INFO:teuthology.orchestra.run.smithi117.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 617 kB/s | 79 kB 00:00 2024-05-15T12:32:11.944 INFO:teuthology.orchestra.run.smithi117.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 174 kB/s | 16 kB 00:00 2024-05-15T12:32:11.970 INFO:teuthology.orchestra.run.smithi117.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.4 MB/s | 272 kB 00:00 2024-05-15T12:32:12.020 INFO:teuthology.orchestra.run.smithi117.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 632 kB/s | 90 kB 00:00 2024-05-15T12:32:12.037 INFO:teuthology.orchestra.run.smithi117.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 389 kB/s | 36 kB 00:00 2024-05-15T12:32:12.087 INFO:teuthology.orchestra.run.smithi117.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-05-15T12:32:12.121 INFO:teuthology.orchestra.run.smithi117.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 239 kB/s | 20 kB 00:00 2024-05-15T12:32:12.254 INFO:teuthology.orchestra.run.smithi117.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.8 MB/s | 427 kB 00:00 2024-05-15T12:32:12.255 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:32:12.256 INFO:teuthology.orchestra.run.smithi117.stdout:Total 9.0 MB/s | 30 MB 00:03 2024-05-15T12:32:12.400 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-05-15T12:32:12.439 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-05-15T12:32:12.439 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-05-15T12:32:12.887 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-05-15T12:32:12.888 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-05-15T12:32:13.687 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-test-2:19.0.0-1882.gae248125.el9.x86_64 4/4 2024-05-15T12:32:13.687 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-test-2:19.0.0-1882.gae248125.el9.x86_64 1/4 2024-05-15T12:32:13.687 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libcephsqlite-2:19.0.0-1882.gae248125.el9.x86_64 2/4 2024-05-15T12:32:13.687 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-05-15T12:32:13.975 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-05-15T12:32:13.976 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:32:13.976 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-05-15T12:32:13.976 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-test-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:32:13.976 INFO:teuthology.orchestra.run.smithi178.stdout: libcephsqlite-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:32:13.976 INFO:teuthology.orchestra.run.smithi178.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-15T12:32:13.976 INFO:teuthology.orchestra.run.smithi178.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-15T12:32:13.976 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:32:13.976 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-05-15T12:32:14.150 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-05-15T12:32:14.223 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-05-15T12:32:14.262 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph 2024-05-15T12:32:14.266 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-05-15T12:32:14.309 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-05-15T12:32:14.381 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-05-15T12:32:14.416 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-05-15T12:32:14.449 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-05-15T12:32:14.541 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-05-15T12:32:14.611 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-05-15T12:32:14.651 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-05-15T12:32:14.687 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-05-15T12:32:14.716 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/36 2024-05-15T12:32:14.751 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/36 2024-05-15T12:32:14.784 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/36 2024-05-15T12:32:14.825 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/36 2024-05-15T12:32:14.862 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/36 2024-05-15T12:32:14.871 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:00:49 ago on Wed 15 May 2024 12:31:25 PM UTC. 2024-05-15T12:32:14.894 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/36 2024-05-15T12:32:14.932 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/36 2024-05-15T12:32:15.004 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-05-15T12:32:15.006 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 18/36 2024-05-15T12:32:15.007 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:32:15.007 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-05-15T12:32:15.007 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:32:15.007 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-05-15T12:32:15.007 INFO:teuthology.orchestra.run.smithi178.stdout: ceph x86_64 2:19.0.0-1882.gae248125.el9 ceph 6.6 k 2024-05-15T12:32:15.007 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-05-15T12:32:15.007 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mds x86_64 2:19.0.0-1882.gae248125.el9 ceph 2.1 M 2024-05-15T12:32:15.007 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr x86_64 2:19.0.0-1882.gae248125.el9 ceph 1.5 M 2024-05-15T12:32:15.007 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1882.gae248125.el9 ceph-noarch 248 k 2024-05-15T12:32:15.007 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mon x86_64 2:19.0.0-1882.gae248125.el9 ceph 4.7 M 2024-05-15T12:32:15.007 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-osd x86_64 2:19.0.0-1882.gae248125.el9 ceph 17 M 2024-05-15T12:32:15.007 INFO:teuthology.orchestra.run.smithi178.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-15T12:32:15.008 INFO:teuthology.orchestra.run.smithi178.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-15T12:32:15.008 INFO:teuthology.orchestra.run.smithi178.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-05-15T12:32:15.008 INFO:teuthology.orchestra.run.smithi178.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-15T12:32:15.008 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-15T12:32:15.008 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-15T12:32:15.008 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-15T12:32:15.008 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-15T12:32:15.008 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-15T12:32:15.008 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-15T12:32:15.008 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-collections 2024-05-15T12:32:15.008 INFO:teuthology.orchestra.run.smithi178.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-05-15T12:32:15.009 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-15T12:32:15.009 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-15T12:32:15.009 INFO:teuthology.orchestra.run.smithi178.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-15T12:32:15.009 INFO:teuthology.orchestra.run.smithi178.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-15T12:32:15.009 INFO:teuthology.orchestra.run.smithi178.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-15T12:32:15.009 INFO:teuthology.orchestra.run.smithi178.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-15T12:32:15.009 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-15T12:32:15.009 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-15T12:32:15.009 INFO:teuthology.orchestra.run.smithi178.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-15T12:32:15.009 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-15T12:32:15.009 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-15T12:32:15.009 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-15T12:32:15.009 INFO:teuthology.orchestra.run.smithi178.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-15T12:32:15.010 INFO:teuthology.orchestra.run.smithi178.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-15T12:32:15.010 INFO:teuthology.orchestra.run.smithi178.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-15T12:32:15.010 INFO:teuthology.orchestra.run.smithi178.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-15T12:32:15.010 INFO:teuthology.orchestra.run.smithi178.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-15T12:32:15.010 INFO:teuthology.orchestra.run.smithi178.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-15T12:32:15.010 INFO:teuthology.orchestra.run.smithi178.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-15T12:32:15.010 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:32:15.010 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-05-15T12:32:15.010 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:32:15.010 INFO:teuthology.orchestra.run.smithi178.stdout:Install 36 Packages 2024-05-15T12:32:15.010 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:32:15.011 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 30 M 2024-05-15T12:32:15.011 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 107 M 2024-05-15T12:32:15.011 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-05-15T12:32:15.152 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 19/36 2024-05-15T12:32:15.240 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 20/36 2024-05-15T12:32:15.294 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 21/36 2024-05-15T12:32:15.332 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 22/36 2024-05-15T12:32:15.389 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-ply-3.11-14.el9.noarch 23/36 2024-05-15T12:32:15.453 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 24/36 2024-05-15T12:32:15.649 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 25/36 2024-05-15T12:32:15.694 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 26/36 2024-05-15T12:32:15.772 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 27/36 2024-05-15T12:32:15.861 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 28/36 2024-05-15T12:32:15.909 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 29/36 2024-05-15T12:32:16.027 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 30/36 2024-05-15T12:32:16.048 INFO:teuthology.orchestra.run.smithi178.stdout:(1/36): ceph-19.0.0-1882.gae248125.el9.x86_64.r 34 kB/s | 6.6 kB 00:00 2024-05-15T12:32:16.208 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1882.gae248125.el9. 31/36 2024-05-15T12:32:16.231 INFO:teuthology.orchestra.run.smithi178.stdout:(2/36): ceph-mds-19.0.0-1882.gae248125.el9.x86_ 5.6 MB/s | 2.1 MB 00:00 2024-05-15T12:32:16.237 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-2:19.0.0-1882.gae248125.el9.x86_64 32/36 2024-05-15T12:32:16.263 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1882.gae248125.el9.x86_64 32/36 2024-05-15T12:32:16.263 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-15T12:32:16.263 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-15T12:32:16.263 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-15T12:32:16.263 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-15T12:32:16.263 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:32:16.273 INFO:teuthology.orchestra.run.smithi178.stdout:(3/36): ceph-mgr-19.0.0-1882.gae248125.el9.x86_ 3.5 MB/s | 1.5 MB 00:00 2024-05-15T12:32:16.382 INFO:teuthology.orchestra.run.smithi178.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1882.gae24 2.2 MB/s | 248 kB 00:00 2024-05-15T12:32:16.532 INFO:teuthology.orchestra.run.smithi178.stdout:(5/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.7 MB/s | 253 kB 00:00 2024-05-15T12:32:16.641 INFO:teuthology.orchestra.run.smithi178.stdout:(6/36): python3-cryptography-36.0.1-4.el9.x86_6 11 MB/s | 1.2 MB 00:00 2024-05-15T12:32:16.682 INFO:teuthology.orchestra.run.smithi178.stdout:(7/36): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-05-15T12:32:16.724 INFO:teuthology.orchestra.run.smithi178.stdout:(8/36): python3-pycparser-2.20-6.el9.noarch.rpm 3.2 MB/s | 135 kB 00:00 2024-05-15T12:32:16.774 INFO:teuthology.orchestra.run.smithi178.stdout:(9/36): python3-requests-2.25.1-8.el9.noarch.rp 2.4 MB/s | 125 kB 00:00 2024-05-15T12:32:16.825 INFO:teuthology.orchestra.run.smithi178.stdout:(10/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.2 MB/s | 215 kB 00:00 2024-05-15T12:32:17.050 INFO:teuthology.orchestra.run.smithi178.stdout:(11/36): ceph-osd-19.0.0-1882.gae248125.el9.x86 21 MB/s | 17 MB 00:00 2024-05-15T12:32:17.109 INFO:teuthology.orchestra.run.smithi178.stdout:(12/36): lua-5.4.4-4.el9.x86_64.rpm 665 kB/s | 188 kB 00:00 2024-05-15T12:32:17.142 INFO:teuthology.orchestra.run.smithi178.stdout:(13/36): python3-markupsafe-1.1.1-12.el9.x86_64 1.0 MB/s | 35 kB 00:00 2024-05-15T12:32:17.176 INFO:teuthology.orchestra.run.smithi178.stdout:(14/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-05-15T12:32:17.201 INFO:teuthology.orchestra.run.smithi178.stdout:(15/36): python3-pytz-2021.1-5.el9.noarch.rpm 874 kB/s | 51 kB 00:00 2024-05-15T12:32:17.226 INFO:teuthology.orchestra.run.smithi178.stdout:(16/36): python3-toml-0.10.2-6.el9.noarch.rpm 832 kB/s | 42 kB 00:00 2024-05-15T12:32:17.368 INFO:teuthology.orchestra.run.smithi178.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 134 kB/s | 22 kB 00:00 2024-05-15T12:32:17.577 INFO:teuthology.orchestra.run.smithi178.stdout:(18/36): ceph-mon-19.0.0-1882.gae248125.el9.x86 3.1 MB/s | 4.7 MB 00:01 2024-05-15T12:32:17.696 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-osd-2:19.0.0-1882.gae248125.el9.x86_64 33/36 2024-05-15T12:32:17.723 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-osd-2:19.0.0-1882.gae248125.el9.x86_64 33/36 2024-05-15T12:32:17.723 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-15T12:32:17.724 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-15T12:32:17.724 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-15T12:32:17.724 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-15T12:32:17.724 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:32:18.163 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mon-2:19.0.0-1882.gae248125.el9.x86_64 34/36 2024-05-15T12:32:18.187 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mon-2:19.0.0-1882.gae248125.el9.x86_64 34/36 2024-05-15T12:32:18.187 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-15T12:32:18.187 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-15T12:32:18.187 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-15T12:32:18.187 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-15T12:32:18.187 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:32:18.389 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mds-2:19.0.0-1882.gae248125.el9.x86_64 35/36 2024-05-15T12:32:18.412 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mds-2:19.0.0-1882.gae248125.el9.x86_64 35/36 2024-05-15T12:32:18.412 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-15T12:32:18.412 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-15T12:32:18.413 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-15T12:32:18.413 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-15T12:32:18.413 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:32:18.478 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-2:19.0.0-1882.gae248125.el9.x86_64 36/36 2024-05-15T12:32:19.036 INFO:teuthology.orchestra.run.smithi178.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 118 kB/s | 172 kB 00:01 2024-05-15T12:32:19.436 INFO:teuthology.orchestra.run.smithi178.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 68 kB/s | 151 kB 00:02 2024-05-15T12:32:20.221 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-2:19.0.0-1882.gae248125.el9.x86_64 36/36 2024-05-15T12:32:20.221 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-2:19.0.0-1882.gae248125.el9.x86_64 1/36 2024-05-15T12:32:20.222 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mds-2:19.0.0-1882.gae248125.el9.x86_64 2/36 2024-05-15T12:32:20.222 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-2:19.0.0-1882.gae248125.el9.x86_64 3/36 2024-05-15T12:32:20.222 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mon-2:19.0.0-1882.gae248125.el9.x86_64 4/36 2024-05-15T12:32:20.222 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-osd-2:19.0.0-1882.gae248125.el9.x86_64 5/36 2024-05-15T12:32:20.222 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1882.gae248125.el9. 6/36 2024-05-15T12:32:20.222 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/36 2024-05-15T12:32:20.222 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/36 2024-05-15T12:32:20.222 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/36 2024-05-15T12:32:20.222 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/36 2024-05-15T12:32:20.222 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/36 2024-05-15T12:32:20.222 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/36 2024-05-15T12:32:20.222 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/36 2024-05-15T12:32:20.222 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/36 2024-05-15T12:32:20.222 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/36 2024-05-15T12:32:20.223 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/36 2024-05-15T12:32:20.223 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/36 2024-05-15T12:32:20.223 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-05-15T12:32:20.223 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-05-15T12:32:20.223 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-05-15T12:32:20.223 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-05-15T12:32:20.223 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-05-15T12:32:20.223 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-05-15T12:32:20.223 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-05-15T12:32:20.223 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-05-15T12:32:20.223 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-05-15T12:32:20.223 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-05-15T12:32:20.223 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-05-15T12:32:20.224 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-05-15T12:32:20.224 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-05-15T12:32:20.224 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-05-15T12:32:20.225 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-05-15T12:32:20.225 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-05-15T12:32:20.225 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-05-15T12:32:20.225 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-05-15T12:32:20.989 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-05-15T12:32:20.989 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:32:20.989 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-05-15T12:32:20.989 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:32:20.990 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mds-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:32:20.990 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:32:20.990 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-modules-core-2:19.0.0-1882.gae248125.el9.noarch 2024-05-15T12:32:20.990 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mon-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:32:20.990 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-osd-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:32:20.990 INFO:teuthology.orchestra.run.smithi117.stdout: lua-5.4.4-4.el9.x86_64 2024-05-15T12:32:20.990 INFO:teuthology.orchestra.run.smithi117.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-15T12:32:20.990 INFO:teuthology.orchestra.run.smithi117.stdout: luarocks-3.9.2-1.el9.noarch 2024-05-15T12:32:20.990 INFO:teuthology.orchestra.run.smithi117.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-15T12:32:20.990 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-15T12:32:20.990 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-15T12:32:20.990 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-15T12:32:20.990 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-15T12:32:20.990 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-15T12:32:20.991 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-15T12:32:20.991 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-15T12:32:20.991 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-15T12:32:20.991 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-15T12:32:20.991 INFO:teuthology.orchestra.run.smithi117.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-15T12:32:20.991 INFO:teuthology.orchestra.run.smithi117.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-15T12:32:20.991 INFO:teuthology.orchestra.run.smithi117.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-15T12:32:20.991 INFO:teuthology.orchestra.run.smithi117.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-15T12:32:20.991 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-15T12:32:20.991 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ply-3.11-14.el9.noarch 2024-05-15T12:32:20.991 INFO:teuthology.orchestra.run.smithi117.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-15T12:32:20.991 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-15T12:32:20.991 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-15T12:32:20.991 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-15T12:32:20.991 INFO:teuthology.orchestra.run.smithi117.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-15T12:32:20.992 INFO:teuthology.orchestra.run.smithi117.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-15T12:32:20.992 INFO:teuthology.orchestra.run.smithi117.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-15T12:32:20.992 INFO:teuthology.orchestra.run.smithi117.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-15T12:32:20.992 INFO:teuthology.orchestra.run.smithi117.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-15T12:32:20.992 INFO:teuthology.orchestra.run.smithi117.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-15T12:32:20.992 INFO:teuthology.orchestra.run.smithi117.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-15T12:32:20.992 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:32:20.992 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-05-15T12:32:21.349 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-base 2024-05-15T12:32:21.578 INFO:teuthology.orchestra.run.smithi178.stdout:(21/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 10 kB/s | 43 kB 00:04 2024-05-15T12:32:21.930 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:00:58 ago on Wed 15 May 2024 12:31:23 PM UTC. 2024-05-15T12:32:22.011 INFO:teuthology.orchestra.run.smithi117.stdout:Package ceph-base-2:19.0.0-1882.gae248125.el9.x86_64 is already installed. 2024-05-15T12:32:22.053 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-05-15T12:32:22.054 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-05-15T12:32:22.054 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-05-15T12:32:22.117 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install cephadm 2024-05-15T12:32:22.386 INFO:teuthology.orchestra.run.smithi178.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 107 kB/s | 358 kB 00:03 2024-05-15T12:32:22.503 INFO:teuthology.orchestra.run.smithi178.stdout:(23/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 3.5 kB/s | 11 kB 00:03 2024-05-15T12:32:22.685 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:00:59 ago on Wed 15 May 2024 12:31:23 PM UTC. 2024-05-15T12:32:22.808 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-05-15T12:32:22.809 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:32:22.809 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-05-15T12:32:22.810 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:32:22.810 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-05-15T12:32:22.810 INFO:teuthology.orchestra.run.smithi117.stdout: cephadm noarch 2:19.0.0-1882.gae248125.el9 ceph-noarch 749 k 2024-05-15T12:32:22.810 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:32:22.810 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-05-15T12:32:22.810 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:32:22.810 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-05-15T12:32:22.810 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:32:22.810 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 749 k 2024-05-15T12:32:22.810 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 755 k 2024-05-15T12:32:22.810 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-05-15T12:32:22.862 INFO:teuthology.orchestra.run.smithi178.stdout:(24/36): python3-jaraco-classes-3.2.1-5.el9.noa 14 kB/s | 18 kB 00:01 2024-05-15T12:32:23.096 INFO:teuthology.orchestra.run.smithi117.stdout:cephadm-19.0.0-1882.gae248125.el9.noarch.rpm 2.6 MB/s | 749 kB 00:00 2024-05-15T12:32:23.097 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:32:23.097 INFO:teuthology.orchestra.run.smithi117.stdout:Total 2.5 MB/s | 749 kB 00:00 2024-05-15T12:32:23.097 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-05-15T12:32:23.102 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-05-15T12:32:23.102 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-05-15T12:32:23.108 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-05-15T12:32:23.108 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-05-15T12:32:23.112 INFO:teuthology.orchestra.run.smithi178.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 32 kB/s | 23 kB 00:00 2024-05-15T12:32:23.183 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-05-15T12:32:23.329 INFO:teuthology.orchestra.run.smithi178.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 24 kB/s | 19 kB 00:00 2024-05-15T12:32:23.554 INFO:teuthology.orchestra.run.smithi178.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 28 kB/s | 20 kB 00:00 2024-05-15T12:32:23.663 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: cephadm-2:19.0.0-1882.gae248125.el9.noarch 1/1 2024-05-15T12:32:23.749 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : cephadm-2:19.0.0-1882.gae248125.el9.noarch 1/1 2024-05-15T12:32:24.046 INFO:teuthology.orchestra.run.smithi178.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 50 kB/s | 46 kB 00:00 2024-05-15T12:32:24.113 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: cephadm-2:19.0.0-1882.gae248125.el9.noarch 1/1 2024-05-15T12:32:24.350 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : cephadm-2:19.0.0-1882.gae248125.el9.noarch 1/1 2024-05-15T12:32:24.351 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:32:24.351 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-05-15T12:32:24.351 INFO:teuthology.orchestra.run.smithi117.stdout: cephadm-2:19.0.0-1882.gae248125.el9.noarch 2024-05-15T12:32:24.351 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:32:24.351 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-05-15T12:32:24.584 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-immutable-object-cache 2024-05-15T12:32:24.721 INFO:teuthology.orchestra.run.smithi178.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 57 kB/s | 79 kB 00:01 2024-05-15T12:32:25.162 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:02 ago on Wed 15 May 2024 12:31:23 PM UTC. 2024-05-15T12:32:25.286 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-05-15T12:32:25.287 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:32:25.287 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-05-15T12:32:25.287 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:32:25.287 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-05-15T12:32:25.287 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1882.gae248125.el9 ceph 145 k 2024-05-15T12:32:25.287 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:32:25.287 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-05-15T12:32:25.287 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:32:25.287 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-05-15T12:32:25.287 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:32:25.287 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 145 k 2024-05-15T12:32:25.287 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 436 k 2024-05-15T12:32:25.288 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-05-15T12:32:25.719 INFO:teuthology.orchestra.run.smithi117.stdout:ceph-immutable-object-cache-19.0.0-1882.gae2481 337 kB/s | 145 kB 00:00 2024-05-15T12:32:25.719 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:32:25.720 INFO:teuthology.orchestra.run.smithi117.stdout:Total 336 kB/s | 145 kB 00:00 2024-05-15T12:32:25.720 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-05-15T12:32:25.727 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-05-15T12:32:25.727 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-05-15T12:32:25.783 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-05-15T12:32:25.783 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-05-15T12:32:25.925 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-05-15T12:32:26.003 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1882.gae248125. 1/1 2024-05-15T12:32:26.028 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1882.gae248125. 1/1 2024-05-15T12:32:26.028 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-15T12:32:26.028 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-15T12:32:26.028 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:32:26.901 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1882.gae248125. 1/1 2024-05-15T12:32:26.901 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:32:26.901 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-05-15T12:32:26.901 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-immutable-object-cache-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:32:26.901 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:32:26.901 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-05-15T12:32:27.172 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-mgr 2024-05-15T12:32:27.429 INFO:teuthology.orchestra.run.smithi178.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 70 kB/s | 272 kB 00:03 2024-05-15T12:32:27.596 INFO:teuthology.orchestra.run.smithi178.stdout:(31/36): python3-portend-3.1.0-2.el9.noarch.rpm 4.6 kB/s | 16 kB 00:03 2024-05-15T12:32:27.756 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:04 ago on Wed 15 May 2024 12:31:23 PM UTC. 2024-05-15T12:32:27.838 INFO:teuthology.orchestra.run.smithi117.stdout:Package ceph-mgr-2:19.0.0-1882.gae248125.el9.x86_64 is already installed. 2024-05-15T12:32:27.881 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-05-15T12:32:27.882 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-05-15T12:32:27.882 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-05-15T12:32:27.955 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-mgr-dashboard 2024-05-15T12:32:28.532 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:05 ago on Wed 15 May 2024 12:31:23 PM UTC. 2024-05-15T12:32:28.655 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-05-15T12:32:28.656 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:32:28.656 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-05-15T12:32:28.656 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:32:28.656 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-05-15T12:32:28.656 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1882.gae248125.el9 ceph-noarch 1.8 M 2024-05-15T12:32:28.656 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-05-15T12:32:28.656 INFO:teuthology.orchestra.run.smithi117.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-15T12:32:28.656 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1882.gae248125.el9 ceph-noarch 31 k 2024-05-15T12:32:28.657 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1882.gae248125.el9 ceph-noarch 16 k 2024-05-15T12:32:28.657 INFO:teuthology.orchestra.run.smithi117.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-15T12:32:28.657 INFO:teuthology.orchestra.run.smithi117.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-15T12:32:28.657 INFO:teuthology.orchestra.run.smithi117.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-15T12:32:28.657 INFO:teuthology.orchestra.run.smithi117.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-15T12:32:28.657 INFO:teuthology.orchestra.run.smithi117.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-15T12:32:28.657 INFO:teuthology.orchestra.run.smithi117.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-15T12:32:28.657 INFO:teuthology.orchestra.run.smithi117.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-15T12:32:28.657 INFO:teuthology.orchestra.run.smithi117.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-15T12:32:28.657 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:32:28.657 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-05-15T12:32:28.657 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:32:28.657 INFO:teuthology.orchestra.run.smithi117.stdout:Install 12 Packages 2024-05-15T12:32:28.657 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:32:28.658 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 6.9 M 2024-05-15T12:32:28.658 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 26 M 2024-05-15T12:32:28.658 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-05-15T12:32:29.296 INFO:teuthology.orchestra.run.smithi178.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 20 kB/s | 90 kB 00:04 2024-05-15T12:32:29.556 INFO:teuthology.orchestra.run.smithi117.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1882.gae2 90 kB/s | 16 kB 00:00 2024-05-15T12:32:29.622 INFO:teuthology.orchestra.run.smithi117.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1882.gae 126 kB/s | 31 kB 00:00 2024-05-15T12:32:29.914 INFO:teuthology.orchestra.run.smithi117.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 923 kB/s | 269 kB 00:00 2024-05-15T12:32:29.938 INFO:teuthology.orchestra.run.smithi178.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 14 kB/s | 36 kB 00:02 2024-05-15T12:32:29.956 INFO:teuthology.orchestra.run.smithi117.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 2.5 MB/s | 1.0 MB 00:00 2024-05-15T12:32:30.149 INFO:teuthology.orchestra.run.smithi117.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.8 MB/s | 548 kB 00:00 2024-05-15T12:32:30.190 INFO:teuthology.orchestra.run.smithi117.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 469 kB/s | 19 kB 00:00 2024-05-15T12:32:30.257 INFO:teuthology.orchestra.run.smithi117.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.5 MB/s | 863 kB 00:00 2024-05-15T12:32:30.299 INFO:teuthology.orchestra.run.smithi117.stdout:(8/12): ceph-mgr-dashboard-19.0.0-1882.gae24812 1.9 MB/s | 1.8 MB 00:00 2024-05-15T12:32:30.341 INFO:teuthology.orchestra.run.smithi117.stdout:(9/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 14 MB/s | 2.0 MB 00:00 2024-05-15T12:32:30.383 INFO:teuthology.orchestra.run.smithi117.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 4.4 MB/s | 188 kB 00:00 2024-05-15T12:32:30.408 INFO:teuthology.orchestra.run.smithi117.stdout:(11/12): python3-repoze-lru-0.7-16.el9.noarch.r 282 kB/s | 31 kB 00:00 2024-05-15T12:32:30.433 INFO:teuthology.orchestra.run.smithi117.stdout:(12/12): python3-grpcio-tools-1.46.7-10.el9.x86 823 kB/s | 144 kB 00:00 2024-05-15T12:32:30.434 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:32:30.435 INFO:teuthology.orchestra.run.smithi117.stdout:Total 3.9 MB/s | 6.9 MB 00:01 2024-05-15T12:32:30.485 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-05-15T12:32:30.501 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-05-15T12:32:30.501 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-05-15T12:32:30.662 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-05-15T12:32:30.663 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-05-15T12:32:31.254 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-05-15T12:32:31.327 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-05-15T12:32:31.361 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-05-15T12:32:31.409 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-05-15T12:32:31.438 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-05-15T12:32:31.542 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-05-15T12:32:31.688 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-05-15T12:32:31.773 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-05-15T12:32:31.843 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-05-15T12:32:31.871 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-05-15T12:32:31.903 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1882.gae248125.el9 10/12 2024-05-15T12:32:32.145 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1882.gae248125.el 11/12 2024-05-15T12:32:32.178 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1882.gae248125.el9.noa 12/12 2024-05-15T12:32:32.894 INFO:teuthology.orchestra.run.smithi178.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 43 kB/s | 230 kB 00:05 2024-05-15T12:32:32.947 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1882.gae248125.el9.noa 12/12 2024-05-15T12:32:32.947 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1882.gae248125.el 1/12 2024-05-15T12:32:32.947 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1882.gae248125.el9.noa 2/12 2024-05-15T12:32:32.947 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1882.gae248125.el9 3/12 2024-05-15T12:32:32.947 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-05-15T12:32:32.947 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-05-15T12:32:32.948 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-05-15T12:32:32.948 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-05-15T12:32:32.948 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-05-15T12:32:32.948 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-05-15T12:32:32.948 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-05-15T12:32:32.948 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-05-15T12:32:33.249 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-05-15T12:32:33.249 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:32:33.250 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-05-15T12:32:33.250 INFO:teuthology.orchestra.run.smithi117.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-15T12:32:33.250 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-grafana-dashboards-2:19.0.0-1882.gae248125.el9.noarch 2024-05-15T12:32:33.250 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-dashboard-2:19.0.0-1882.gae248125.el9.noarch 2024-05-15T12:32:33.250 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-prometheus-alerts-2:19.0.0-1882.gae248125.el9.noarch 2024-05-15T12:32:33.250 INFO:teuthology.orchestra.run.smithi117.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-15T12:32:33.250 INFO:teuthology.orchestra.run.smithi117.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-15T12:32:33.250 INFO:teuthology.orchestra.run.smithi117.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-15T12:32:33.250 INFO:teuthology.orchestra.run.smithi117.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-15T12:32:33.250 INFO:teuthology.orchestra.run.smithi117.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-15T12:32:33.250 INFO:teuthology.orchestra.run.smithi117.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-15T12:32:33.250 INFO:teuthology.orchestra.run.smithi117.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-15T12:32:33.251 INFO:teuthology.orchestra.run.smithi117.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-15T12:32:33.251 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:32:33.251 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-05-15T12:32:33.485 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-mgr-diskprediction-local 2024-05-15T12:32:34.053 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:11 ago on Wed 15 May 2024 12:31:23 PM UTC. 2024-05-15T12:32:34.174 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-05-15T12:32:34.175 INFO:teuthology.orchestra.run.smithi117.stdout:======================================================================================= 2024-05-15T12:32:34.176 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-05-15T12:32:34.176 INFO:teuthology.orchestra.run.smithi117.stdout:======================================================================================= 2024-05-15T12:32:34.176 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-05-15T12:32:34.176 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1882.gae248125.el9 ceph-noarch 7.4 M 2024-05-15T12:32:34.176 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-05-15T12:32:34.176 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-15T12:32:34.176 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-15T12:32:34.176 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-15T12:32:34.176 INFO:teuthology.orchestra.run.smithi117.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-15T12:32:34.176 INFO:teuthology.orchestra.run.smithi117.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-15T12:32:34.176 INFO:teuthology.orchestra.run.smithi117.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-15T12:32:34.176 INFO:teuthology.orchestra.run.smithi117.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-15T12:32:34.176 INFO:teuthology.orchestra.run.smithi117.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-15T12:32:34.176 INFO:teuthology.orchestra.run.smithi117.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-15T12:32:34.176 INFO:teuthology.orchestra.run.smithi117.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-15T12:32:34.176 INFO:teuthology.orchestra.run.smithi117.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-15T12:32:34.177 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:32:34.177 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-05-15T12:32:34.177 INFO:teuthology.orchestra.run.smithi117.stdout:======================================================================================= 2024-05-15T12:32:34.177 INFO:teuthology.orchestra.run.smithi117.stdout:Install 12 Packages 2024-05-15T12:32:34.177 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:32:34.177 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 38 M 2024-05-15T12:32:34.177 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 209 M 2024-05-15T12:32:34.177 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-05-15T12:32:34.731 INFO:teuthology.orchestra.run.smithi117.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.4 MB/s | 195 kB 00:00 2024-05-15T12:32:34.782 INFO:teuthology.orchestra.run.smithi117.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.2 MB/s | 803 kB 00:00 2024-05-15T12:32:34.857 INFO:teuthology.orchestra.run.smithi117.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 261 kB/s | 33 kB 00:00 2024-05-15T12:32:34.899 INFO:teuthology.orchestra.run.smithi117.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 436 kB/s | 18 kB 00:00 2024-05-15T12:32:34.940 INFO:teuthology.orchestra.run.smithi117.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 850 kB/s | 35 kB 00:00 2024-05-15T12:32:35.074 INFO:teuthology.orchestra.run.smithi117.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 10 MB/s | 3.0 MB 00:00 2024-05-15T12:32:35.124 INFO:teuthology.orchestra.run.smithi117.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 4.8 MB/s | 244 kB 00:00 2024-05-15T12:32:35.216 INFO:teuthology.orchestra.run.smithi117.stdout:(8/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 17 MB/s | 4.7 MB 00:00 2024-05-15T12:32:35.275 INFO:teuthology.orchestra.run.smithi117.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 4.2 MB/s | 250 kB 00:00 2024-05-15T12:32:35.351 INFO:teuthology.orchestra.run.smithi117.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 23 MB/s | 5.1 MB 00:00 2024-05-15T12:32:35.651 INFO:teuthology.orchestra.run.smithi117.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 43 MB/s | 16 MB 00:00 2024-05-15T12:32:35.793 INFO:teuthology.orchestra.run.smithi117.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 6.2 MB/s | 7.4 MB 00:01 2024-05-15T12:32:35.794 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:32:35.794 INFO:teuthology.orchestra.run.smithi117.stdout:Total 24 MB/s | 38 MB 00:01 2024-05-15T12:32:36.062 INFO:teuthology.orchestra.run.smithi178.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 3.3 kB/s | 20 kB 00:06 2024-05-15T12:32:36.087 INFO:teuthology.orchestra.run.smithi178.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 63 kB/s | 427 kB 00:06 2024-05-15T12:32:36.093 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:32:36.093 INFO:teuthology.orchestra.run.smithi178.stdout:Total 1.4 MB/s | 30 MB 00:21 2024-05-15T12:32:36.145 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-05-15T12:32:36.170 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-05-15T12:32:36.170 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-05-15T12:32:36.231 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-05-15T12:32:36.271 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-05-15T12:32:36.271 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-05-15T12:32:36.603 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-05-15T12:32:36.603 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-05-15T12:32:36.711 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-05-15T12:32:36.712 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-05-15T12:32:37.353 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-05-15T12:32:37.448 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-05-15T12:32:37.490 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-05-15T12:32:37.589 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-05-15T12:32:37.625 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-05-15T12:32:37.802 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-05-15T12:32:37.832 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-05-15T12:32:37.908 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-05-15T12:32:37.959 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-05-15T12:32:38.031 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-05-15T12:32:38.064 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-05-15T12:32:38.100 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-05-15T12:32:38.180 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-05-15T12:32:38.224 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-05-15T12:32:38.257 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-05-15T12:32:38.350 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-05-15T12:32:38.427 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-05-15T12:32:38.468 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-05-15T12:32:38.503 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-05-15T12:32:38.533 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/36 2024-05-15T12:32:38.559 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/36 2024-05-15T12:32:38.590 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-05-15T12:32:38.592 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/36 2024-05-15T12:32:38.633 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/36 2024-05-15T12:32:38.685 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/36 2024-05-15T12:32:38.710 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-05-15T12:32:38.719 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/36 2024-05-15T12:32:38.757 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/36 2024-05-15T12:32:38.830 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 18/36 2024-05-15T12:32:38.988 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 19/36 2024-05-15T12:32:39.067 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 20/36 2024-05-15T12:32:39.120 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 21/36 2024-05-15T12:32:39.159 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 22/36 2024-05-15T12:32:39.208 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-ply-3.11-14.el9.noarch 23/36 2024-05-15T12:32:39.271 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 24/36 2024-05-15T12:32:39.467 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 25/36 2024-05-15T12:32:39.514 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 26/36 2024-05-15T12:32:39.584 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 27/36 2024-05-15T12:32:39.672 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 28/36 2024-05-15T12:32:39.704 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 29/36 2024-05-15T12:32:39.821 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 30/36 2024-05-15T12:32:39.928 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-05-15T12:32:39.986 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1882.gae248125.el9. 31/36 2024-05-15T12:32:40.015 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-2:19.0.0-1882.gae248125.el9.x86_64 32/36 2024-05-15T12:32:40.042 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1882.gae248125.el9.x86_64 32/36 2024-05-15T12:32:40.042 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-15T12:32:40.042 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-15T12:32:40.042 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-15T12:32:40.042 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-15T12:32:40.042 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:32:40.963 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-05-15T12:32:40.989 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1882.gae248 12/12 2024-05-15T12:32:41.482 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-osd-2:19.0.0-1882.gae248125.el9.x86_64 33/36 2024-05-15T12:32:41.510 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-osd-2:19.0.0-1882.gae248125.el9.x86_64 33/36 2024-05-15T12:32:41.510 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-15T12:32:41.510 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-15T12:32:41.510 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-15T12:32:41.510 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-15T12:32:41.510 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:32:41.950 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mon-2:19.0.0-1882.gae248125.el9.x86_64 34/36 2024-05-15T12:32:41.974 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mon-2:19.0.0-1882.gae248125.el9.x86_64 34/36 2024-05-15T12:32:41.974 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-15T12:32:41.974 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-15T12:32:41.974 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-15T12:32:41.975 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-15T12:32:41.975 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:32:42.175 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mds-2:19.0.0-1882.gae248125.el9.x86_64 35/36 2024-05-15T12:32:42.199 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mds-2:19.0.0-1882.gae248125.el9.x86_64 35/36 2024-05-15T12:32:42.200 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-15T12:32:42.200 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-15T12:32:42.200 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-15T12:32:42.200 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-15T12:32:42.200 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:32:42.265 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-2:19.0.0-1882.gae248125.el9.x86_64 36/36 2024-05-15T12:32:43.513 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1882.gae248 12/12 2024-05-15T12:32:43.513 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1882.gae248 1/12 2024-05-15T12:32:43.513 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-05-15T12:32:43.513 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-05-15T12:32:43.514 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-05-15T12:32:43.514 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-05-15T12:32:43.514 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-05-15T12:32:43.514 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-05-15T12:32:43.514 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-05-15T12:32:43.514 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-05-15T12:32:43.514 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-05-15T12:32:43.514 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-05-15T12:32:43.888 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-05-15T12:32:43.888 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:32:43.888 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-05-15T12:32:43.888 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1882.gae248125.el9.noarch 2024-05-15T12:32:43.888 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-15T12:32:43.888 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-15T12:32:43.888 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-15T12:32:43.888 INFO:teuthology.orchestra.run.smithi117.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-15T12:32:43.889 INFO:teuthology.orchestra.run.smithi117.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-15T12:32:43.889 INFO:teuthology.orchestra.run.smithi117.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-15T12:32:43.889 INFO:teuthology.orchestra.run.smithi117.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-15T12:32:43.889 INFO:teuthology.orchestra.run.smithi117.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-15T12:32:43.889 INFO:teuthology.orchestra.run.smithi117.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-15T12:32:43.889 INFO:teuthology.orchestra.run.smithi117.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-15T12:32:43.889 INFO:teuthology.orchestra.run.smithi117.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-15T12:32:43.889 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:32:43.889 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-05-15T12:32:43.943 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-2:19.0.0-1882.gae248125.el9.x86_64 36/36 2024-05-15T12:32:43.943 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-2:19.0.0-1882.gae248125.el9.x86_64 1/36 2024-05-15T12:32:43.943 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mds-2:19.0.0-1882.gae248125.el9.x86_64 2/36 2024-05-15T12:32:43.943 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-2:19.0.0-1882.gae248125.el9.x86_64 3/36 2024-05-15T12:32:43.944 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mon-2:19.0.0-1882.gae248125.el9.x86_64 4/36 2024-05-15T12:32:43.944 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-osd-2:19.0.0-1882.gae248125.el9.x86_64 5/36 2024-05-15T12:32:43.944 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1882.gae248125.el9. 6/36 2024-05-15T12:32:43.944 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/36 2024-05-15T12:32:43.944 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/36 2024-05-15T12:32:43.944 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/36 2024-05-15T12:32:43.944 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/36 2024-05-15T12:32:43.944 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/36 2024-05-15T12:32:43.944 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/36 2024-05-15T12:32:43.944 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/36 2024-05-15T12:32:43.944 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/36 2024-05-15T12:32:43.944 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/36 2024-05-15T12:32:43.944 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/36 2024-05-15T12:32:43.944 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/36 2024-05-15T12:32:43.944 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-05-15T12:32:43.945 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-05-15T12:32:43.945 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-05-15T12:32:43.945 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-05-15T12:32:43.945 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-05-15T12:32:43.945 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-05-15T12:32:43.945 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-05-15T12:32:43.945 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-05-15T12:32:43.945 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-05-15T12:32:43.945 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-05-15T12:32:43.945 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-05-15T12:32:43.945 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-05-15T12:32:43.945 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-05-15T12:32:43.945 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-05-15T12:32:43.945 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-05-15T12:32:43.945 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-05-15T12:32:43.946 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-05-15T12:32:43.946 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-05-15T12:32:44.198 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-mgr-rook 2024-05-15T12:32:44.785 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:21 ago on Wed 15 May 2024 12:31:23 PM UTC. 2024-05-15T12:32:44.831 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-05-15T12:32:44.832 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:32:44.832 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-05-15T12:32:44.832 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:32:44.832 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mds-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:32:44.832 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:32:44.832 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-modules-core-2:19.0.0-1882.gae248125.el9.noarch 2024-05-15T12:32:44.832 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mon-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:32:44.832 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-osd-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:32:44.832 INFO:teuthology.orchestra.run.smithi178.stdout: lua-5.4.4-4.el9.x86_64 2024-05-15T12:32:44.832 INFO:teuthology.orchestra.run.smithi178.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-15T12:32:44.832 INFO:teuthology.orchestra.run.smithi178.stdout: luarocks-3.9.2-1.el9.noarch 2024-05-15T12:32:44.832 INFO:teuthology.orchestra.run.smithi178.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-15T12:32:44.832 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-15T12:32:44.832 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-15T12:32:44.832 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-15T12:32:44.833 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-15T12:32:44.833 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-15T12:32:44.833 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-15T12:32:44.833 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-15T12:32:44.833 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-15T12:32:44.833 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-15T12:32:44.833 INFO:teuthology.orchestra.run.smithi178.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-15T12:32:44.833 INFO:teuthology.orchestra.run.smithi178.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-15T12:32:44.833 INFO:teuthology.orchestra.run.smithi178.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-15T12:32:44.833 INFO:teuthology.orchestra.run.smithi178.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-15T12:32:44.833 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-15T12:32:44.833 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ply-3.11-14.el9.noarch 2024-05-15T12:32:44.833 INFO:teuthology.orchestra.run.smithi178.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-15T12:32:44.833 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-15T12:32:44.833 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-15T12:32:44.834 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-15T12:32:44.834 INFO:teuthology.orchestra.run.smithi178.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-15T12:32:44.834 INFO:teuthology.orchestra.run.smithi178.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-15T12:32:44.834 INFO:teuthology.orchestra.run.smithi178.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-15T12:32:44.834 INFO:teuthology.orchestra.run.smithi178.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-15T12:32:44.834 INFO:teuthology.orchestra.run.smithi178.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-15T12:32:44.834 INFO:teuthology.orchestra.run.smithi178.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-15T12:32:44.834 INFO:teuthology.orchestra.run.smithi178.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-15T12:32:44.834 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:32:44.834 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-05-15T12:32:44.907 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-05-15T12:32:44.908 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:32:44.908 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-05-15T12:32:44.908 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:32:44.909 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-05-15T12:32:44.909 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-rook noarch 2:19.0.0-1882.gae248125.el9 ceph-noarch 49 k 2024-05-15T12:32:44.909 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-05-15T12:32:44.909 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-15T12:32:44.909 INFO:teuthology.orchestra.run.smithi117.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-15T12:32:44.909 INFO:teuthology.orchestra.run.smithi117.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-15T12:32:44.909 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-15T12:32:44.909 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-15T12:32:44.909 INFO:teuthology.orchestra.run.smithi117.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-15T12:32:44.909 INFO:teuthology.orchestra.run.smithi117.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-15T12:32:44.909 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-15T12:32:44.909 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-15T12:32:44.909 INFO:teuthology.orchestra.run.smithi117.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-15T12:32:44.909 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-15T12:32:44.909 INFO:teuthology.orchestra.run.smithi117.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-15T12:32:44.910 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:32:44.910 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-05-15T12:32:44.910 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:32:44.910 INFO:teuthology.orchestra.run.smithi117.stdout:Install 13 Packages 2024-05-15T12:32:44.910 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:32:44.911 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 2.2 M 2024-05-15T12:32:44.911 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 26 M 2024-05-15T12:32:44.911 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-05-15T12:32:45.190 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-base 2024-05-15T12:32:45.330 INFO:teuthology.orchestra.run.smithi117.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 290 kB/s | 26 kB 00:00 2024-05-15T12:32:45.347 INFO:teuthology.orchestra.run.smithi117.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 182 kB/s | 19 kB 00:00 2024-05-15T12:32:45.439 INFO:teuthology.orchestra.run.smithi117.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.4 MB/s | 222 kB 00:00 2024-05-15T12:32:45.464 INFO:teuthology.orchestra.run.smithi117.stdout:(4/13): ceph-mgr-rook-19.0.0-1882.gae248125.el9 221 kB/s | 49 kB 00:00 2024-05-15T12:32:45.490 INFO:teuthology.orchestra.run.smithi117.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.1 MB/s | 159 kB 00:00 2024-05-15T12:32:45.523 INFO:teuthology.orchestra.run.smithi117.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 918 kB/s | 54 kB 00:00 2024-05-15T12:32:45.549 INFO:teuthology.orchestra.run.smithi117.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.5 MB/s | 279 kB 00:00 2024-05-15T12:32:45.774 INFO:teuthology.orchestra.run.smithi117.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 60 kB/s | 15 kB 00:00 2024-05-15T12:32:45.796 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:20 ago on Wed 15 May 2024 12:31:25 PM UTC. 2024-05-15T12:32:45.799 INFO:teuthology.orchestra.run.smithi117.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 104 kB/s | 32 kB 00:00 2024-05-15T12:32:45.881 INFO:teuthology.orchestra.run.smithi178.stdout:Package ceph-base-2:19.0.0-1882.gae248125.el9.x86_64 is already installed. 2024-05-15T12:32:45.926 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-05-15T12:32:45.927 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-05-15T12:32:45.927 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-05-15T12:32:45.998 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install cephadm 2024-05-15T12:32:46.016 INFO:teuthology.orchestra.run.smithi117.stdout:(10/13): python3-google-auth-2.29.0-1.el9.noarc 440 kB/s | 206 kB 00:00 2024-05-15T12:32:46.117 INFO:teuthology.orchestra.run.smithi117.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 186 kB/s | 59 kB 00:00 2024-05-15T12:32:46.416 INFO:teuthology.orchestra.run.smithi117.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 224 kB/s | 90 kB 00:00 2024-05-15T12:32:46.567 INFO:teuthology.orchestra.run.smithi117.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.3 MB/s | 1.0 MB 00:00 2024-05-15T12:32:46.568 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:32:46.568 INFO:teuthology.orchestra.run.smithi117.stdout:Total 1.3 MB/s | 2.2 MB 00:01 2024-05-15T12:32:46.577 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:21 ago on Wed 15 May 2024 12:31:25 PM UTC. 2024-05-15T12:32:46.639 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-05-15T12:32:46.654 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-05-15T12:32:46.654 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-05-15T12:32:46.700 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-05-15T12:32:46.700 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:32:46.701 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-05-15T12:32:46.701 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:32:46.701 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-05-15T12:32:46.701 INFO:teuthology.orchestra.run.smithi178.stdout: cephadm noarch 2:19.0.0-1882.gae248125.el9 ceph-noarch 749 k 2024-05-15T12:32:46.701 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:32:46.701 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-05-15T12:32:46.701 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:32:46.701 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-05-15T12:32:46.701 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:32:46.701 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 749 k 2024-05-15T12:32:46.701 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 755 k 2024-05-15T12:32:46.701 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-05-15T12:32:46.740 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-05-15T12:32:46.741 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-05-15T12:32:47.135 INFO:teuthology.orchestra.run.smithi178.stdout:cephadm-19.0.0-1882.gae248125.el9.noarch.rpm 1.7 MB/s | 749 kB 00:00 2024-05-15T12:32:47.136 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:32:47.136 INFO:teuthology.orchestra.run.smithi178.stdout:Total 1.7 MB/s | 749 kB 00:00 2024-05-15T12:32:47.136 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-05-15T12:32:47.141 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-05-15T12:32:47.142 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-05-15T12:32:47.147 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-05-15T12:32:47.148 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-05-15T12:32:47.222 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-05-15T12:32:47.250 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-05-15T12:32:47.423 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-05-15T12:32:47.483 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-05-15T12:32:47.539 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-05-15T12:32:47.584 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-05-15T12:32:47.625 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-05-15T12:32:47.670 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: cephadm-2:19.0.0-1882.gae248125.el9.noarch 1/1 2024-05-15T12:32:47.719 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-05-15T12:32:47.765 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : cephadm-2:19.0.0-1882.gae248125.el9.noarch 1/1 2024-05-15T12:32:47.829 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 7/13 2024-05-15T12:32:47.877 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-05-15T12:32:48.136 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: cephadm-2:19.0.0-1882.gae248125.el9.noarch 1/1 2024-05-15T12:32:48.404 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : cephadm-2:19.0.0-1882.gae248125.el9.noarch 1/1 2024-05-15T12:32:48.404 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:32:48.404 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-05-15T12:32:48.404 INFO:teuthology.orchestra.run.smithi178.stdout: cephadm-2:19.0.0-1882.gae248125.el9.noarch 2024-05-15T12:32:48.404 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:32:48.404 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-05-15T12:32:48.604 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-05-15T12:32:48.640 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-immutable-object-cache 2024-05-15T12:32:48.666 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-05-15T12:32:48.700 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-05-15T12:32:48.748 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-05-15T12:32:48.778 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-rook-2:19.0.0-1882.gae248125.el9.noarch 13/13 2024-05-15T12:32:49.222 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:24 ago on Wed 15 May 2024 12:31:25 PM UTC. 2024-05-15T12:32:49.350 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-05-15T12:32:49.350 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:32:49.350 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-05-15T12:32:49.351 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:32:49.351 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-05-15T12:32:49.351 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1882.gae248125.el9 ceph 145 k 2024-05-15T12:32:49.351 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:32:49.351 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-05-15T12:32:49.351 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:32:49.351 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-05-15T12:32:49.351 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:32:49.352 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 145 k 2024-05-15T12:32:49.352 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 436 k 2024-05-15T12:32:49.352 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-05-15T12:32:49.555 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1882.gae248125.el9.noarch 13/13 2024-05-15T12:32:49.555 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1882.gae248125.el9.noarch 1/13 2024-05-15T12:32:49.555 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-05-15T12:32:49.555 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-05-15T12:32:49.555 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-05-15T12:32:49.555 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-05-15T12:32:49.555 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-05-15T12:32:49.555 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-05-15T12:32:49.555 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-05-15T12:32:49.555 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-05-15T12:32:49.555 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 10/13 2024-05-15T12:32:49.555 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-05-15T12:32:49.555 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-05-15T12:32:49.576 INFO:teuthology.orchestra.run.smithi178.stdout:ceph-immutable-object-cache-19.0.0-1882.gae2481 648 kB/s | 145 kB 00:00 2024-05-15T12:32:49.577 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:32:49.577 INFO:teuthology.orchestra.run.smithi178.stdout:Total 643 kB/s | 145 kB 00:00 2024-05-15T12:32:49.577 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-05-15T12:32:49.584 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-05-15T12:32:49.584 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-05-15T12:32:49.641 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-05-15T12:32:49.641 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-05-15T12:32:49.792 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-05-15T12:32:49.853 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-05-15T12:32:49.853 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:32:49.853 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-05-15T12:32:49.853 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-rook-2:19.0.0-1882.gae248125.el9.noarch 2024-05-15T12:32:49.854 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-15T12:32:49.854 INFO:teuthology.orchestra.run.smithi117.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-15T12:32:49.854 INFO:teuthology.orchestra.run.smithi117.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-15T12:32:49.854 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-15T12:32:49.854 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-15T12:32:49.854 INFO:teuthology.orchestra.run.smithi117.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-15T12:32:49.854 INFO:teuthology.orchestra.run.smithi117.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-15T12:32:49.854 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-15T12:32:49.854 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-15T12:32:49.854 INFO:teuthology.orchestra.run.smithi117.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-15T12:32:49.854 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-15T12:32:49.854 INFO:teuthology.orchestra.run.smithi117.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-15T12:32:49.854 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:32:49.854 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-05-15T12:32:49.860 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1882.gae248125. 1/1 2024-05-15T12:32:49.885 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1882.gae248125. 1/1 2024-05-15T12:32:49.885 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-15T12:32:49.886 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-15T12:32:49.886 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:32:50.125 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-mgr-cephadm 2024-05-15T12:32:50.701 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:27 ago on Wed 15 May 2024 12:31:23 PM UTC. 2024-05-15T12:32:50.802 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1882.gae248125. 1/1 2024-05-15T12:32:50.802 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:32:50.803 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-05-15T12:32:50.803 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-immutable-object-cache-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:32:50.803 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:32:50.803 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-05-15T12:32:50.823 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-05-15T12:32:50.824 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:32:50.824 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-05-15T12:32:50.824 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:32:50.824 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-05-15T12:32:50.825 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1882.gae248125.el9 ceph-noarch 138 k 2024-05-15T12:32:50.825 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-05-15T12:32:50.825 INFO:teuthology.orchestra.run.smithi117.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-15T12:32:50.825 INFO:teuthology.orchestra.run.smithi117.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-15T12:32:50.825 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-05-15T12:32:50.825 INFO:teuthology.orchestra.run.smithi117.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-15T12:32:50.825 INFO:teuthology.orchestra.run.smithi117.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-15T12:32:50.825 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:32:50.826 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-05-15T12:32:50.826 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:32:50.826 INFO:teuthology.orchestra.run.smithi117.stdout:Install 6 Packages 2024-05-15T12:32:50.826 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:32:50.827 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 7.0 M 2024-05-15T12:32:50.827 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 33 M 2024-05-15T12:32:50.827 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-05-15T12:32:51.044 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-mgr 2024-05-15T12:32:51.509 INFO:teuthology.orchestra.run.smithi117.stdout:(1/6): ceph-mgr-cephadm-19.0.0-1882.gae248125.e 521 kB/s | 138 kB 00:00 2024-05-15T12:32:51.534 INFO:teuthology.orchestra.run.smithi117.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 853 kB/s | 248 kB 00:00 2024-05-15T12:32:51.626 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:26 ago on Wed 15 May 2024 12:31:25 PM UTC. 2024-05-15T12:32:51.707 INFO:teuthology.orchestra.run.smithi178.stdout:Package ceph-mgr-2:19.0.0-1882.gae248125.el9.x86_64 is already installed. 2024-05-15T12:32:51.709 INFO:teuthology.orchestra.run.smithi117.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 330 kB/s | 58 kB 00:00 2024-05-15T12:32:51.749 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-05-15T12:32:51.750 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-05-15T12:32:51.750 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-05-15T12:32:51.820 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-mgr-dashboard 2024-05-15T12:32:51.851 INFO:teuthology.orchestra.run.smithi117.stdout:(4/6): python3-babel-2.9.1-2.el9.noarch.rpm 9.8 MB/s | 6.0 MB 00:00 2024-05-15T12:32:51.876 INFO:teuthology.orchestra.run.smithi117.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.5 MB/s | 548 kB 00:00 2024-05-15T12:32:51.901 INFO:teuthology.orchestra.run.smithi117.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 266 kB/s | 51 kB 00:00 2024-05-15T12:32:51.903 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:32:51.903 INFO:teuthology.orchestra.run.smithi117.stdout:Total 6.5 MB/s | 7.0 MB 00:01 2024-05-15T12:32:51.975 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-05-15T12:32:51.986 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-05-15T12:32:51.986 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-05-15T12:32:52.082 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-05-15T12:32:52.083 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-05-15T12:32:52.380 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-05-15T12:32:52.402 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:27 ago on Wed 15 May 2024 12:31:25 PM UTC. 2024-05-15T12:32:52.527 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-05-15T12:32:52.528 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:32:52.528 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-05-15T12:32:52.529 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:32:52.529 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-05-15T12:32:52.529 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1882.gae248125.el9 ceph-noarch 1.8 M 2024-05-15T12:32:52.529 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-05-15T12:32:52.529 INFO:teuthology.orchestra.run.smithi178.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-15T12:32:52.529 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1882.gae248125.el9 ceph-noarch 31 k 2024-05-15T12:32:52.529 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1882.gae248125.el9 ceph-noarch 16 k 2024-05-15T12:32:52.529 INFO:teuthology.orchestra.run.smithi178.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-15T12:32:52.529 INFO:teuthology.orchestra.run.smithi178.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-15T12:32:52.529 INFO:teuthology.orchestra.run.smithi178.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-15T12:32:52.529 INFO:teuthology.orchestra.run.smithi178.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-15T12:32:52.529 INFO:teuthology.orchestra.run.smithi178.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-15T12:32:52.529 INFO:teuthology.orchestra.run.smithi178.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-15T12:32:52.529 INFO:teuthology.orchestra.run.smithi178.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-15T12:32:52.530 INFO:teuthology.orchestra.run.smithi178.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-15T12:32:52.530 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:32:52.530 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-05-15T12:32:52.530 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:32:52.530 INFO:teuthology.orchestra.run.smithi178.stdout:Install 12 Packages 2024-05-15T12:32:52.530 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:32:52.530 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 6.9 M 2024-05-15T12:32:52.530 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 26 M 2024-05-15T12:32:52.531 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-05-15T12:32:52.549 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-05-15T12:32:52.611 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-05-15T12:32:53.139 INFO:teuthology.orchestra.run.smithi178.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1882.gae 124 kB/s | 31 kB 00:00 2024-05-15T12:32:53.150 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-05-15T12:32:53.164 INFO:teuthology.orchestra.run.smithi178.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1882.gae2 59 kB/s | 16 kB 00:00 2024-05-15T12:32:53.230 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-05-15T12:32:53.293 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-05-15T12:32:53.306 INFO:teuthology.orchestra.run.smithi178.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.9 MB/s | 269 kB 00:00 2024-05-15T12:32:53.316 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1882.gae248125.el9.noarch 6/6 2024-05-15T12:32:53.364 INFO:teuthology.orchestra.run.smithi178.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.5 MB/s | 1.0 MB 00:00 2024-05-15T12:32:53.448 INFO:teuthology.orchestra.run.smithi178.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1882.gae24812 3.2 MB/s | 1.8 MB 00:00 2024-05-15T12:32:53.482 INFO:teuthology.orchestra.run.smithi178.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 4.8 MB/s | 863 kB 00:00 2024-05-15T12:32:53.557 INFO:teuthology.orchestra.run.smithi178.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 179 kB/s | 19 kB 00:00 2024-05-15T12:32:53.607 INFO:teuthology.orchestra.run.smithi178.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.2 MB/s | 548 kB 00:00 2024-05-15T12:32:53.708 INFO:teuthology.orchestra.run.smithi178.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 307 kB/s | 31 kB 00:00 2024-05-15T12:32:53.733 INFO:teuthology.orchestra.run.smithi178.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 823 kB/s | 144 kB 00:00 2024-05-15T12:32:53.775 INFO:teuthology.orchestra.run.smithi178.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 7.0 MB/s | 2.0 MB 00:00 2024-05-15T12:32:53.800 INFO:teuthology.orchestra.run.smithi178.stdout:(12/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.0 MB/s | 188 kB 00:00 2024-05-15T12:32:53.802 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:32:53.802 INFO:teuthology.orchestra.run.smithi178.stdout:Total 5.4 MB/s | 6.9 MB 00:01 2024-05-15T12:32:53.851 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-05-15T12:32:53.867 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-05-15T12:32:53.868 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-05-15T12:32:54.030 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-05-15T12:32:54.030 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-05-15T12:32:54.133 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1882.gae248125.el9.noarch 6/6 2024-05-15T12:32:54.133 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1882.gae248125.el9.noarch 1/6 2024-05-15T12:32:54.133 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-05-15T12:32:54.133 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-05-15T12:32:54.133 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-05-15T12:32:54.133 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-05-15T12:32:54.393 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-05-15T12:32:54.393 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:32:54.393 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-05-15T12:32:54.394 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-cephadm-2:19.0.0-1882.gae248125.el9.noarch 2024-05-15T12:32:54.394 INFO:teuthology.orchestra.run.smithi117.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-15T12:32:54.394 INFO:teuthology.orchestra.run.smithi117.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-15T12:32:54.394 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-05-15T12:32:54.394 INFO:teuthology.orchestra.run.smithi117.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-15T12:32:54.394 INFO:teuthology.orchestra.run.smithi117.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-15T12:32:54.394 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:32:54.394 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-05-15T12:32:54.624 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-fuse 2024-05-15T12:32:54.664 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-05-15T12:32:54.743 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-05-15T12:32:54.778 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-05-15T12:32:54.825 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-05-15T12:32:54.855 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-05-15T12:32:54.963 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-05-15T12:32:55.092 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-05-15T12:32:55.185 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-05-15T12:32:55.196 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:32 ago on Wed 15 May 2024 12:31:23 PM UTC. 2024-05-15T12:32:55.237 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-05-15T12:32:55.264 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-05-15T12:32:55.296 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1882.gae248125.el9 10/12 2024-05-15T12:32:55.318 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-05-15T12:32:55.318 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:32:55.318 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-05-15T12:32:55.319 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:32:55.319 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-05-15T12:32:55.319 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-fuse x86_64 2:19.0.0-1882.gae248125.el9 ceph 888 k 2024-05-15T12:32:55.319 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-05-15T12:32:55.319 INFO:teuthology.orchestra.run.smithi117.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-15T12:32:55.319 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:32:55.319 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-05-15T12:32:55.319 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:32:55.319 INFO:teuthology.orchestra.run.smithi117.stdout:Install 2 Packages 2024-05-15T12:32:55.319 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:32:55.320 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 968 k 2024-05-15T12:32:55.320 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 2.8 M 2024-05-15T12:32:55.320 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-05-15T12:32:55.544 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1882.gae248125.el 11/12 2024-05-15T12:32:55.570 INFO:teuthology.orchestra.run.smithi117.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 612 kB/s | 80 kB 00:00 2024-05-15T12:32:55.571 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1882.gae248125.el9.noa 12/12 2024-05-15T12:32:55.745 INFO:teuthology.orchestra.run.smithi117.stdout:(2/2): ceph-fuse-19.0.0-1882.gae248125.el9.x86_ 2.8 MB/s | 888 kB 00:00 2024-05-15T12:32:55.746 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:32:55.746 INFO:teuthology.orchestra.run.smithi117.stdout:Total 2.2 MB/s | 968 kB 00:00 2024-05-15T12:32:55.758 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-05-15T12:32:55.780 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-05-15T12:32:55.780 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-05-15T12:32:55.857 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-05-15T12:32:55.857 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-05-15T12:32:56.049 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-05-15T12:32:56.180 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-05-15T12:32:56.244 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-fuse-2:19.0.0-1882.gae248125.el9.x86_64 2/2 2024-05-15T12:32:56.357 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1882.gae248125.el9.noa 12/12 2024-05-15T12:32:56.357 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1882.gae248125.el 1/12 2024-05-15T12:32:56.357 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1882.gae248125.el9.noa 2/12 2024-05-15T12:32:56.357 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1882.gae248125.el9 3/12 2024-05-15T12:32:56.357 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-05-15T12:32:56.357 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-05-15T12:32:56.357 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-05-15T12:32:56.357 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-05-15T12:32:56.357 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-05-15T12:32:56.357 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-05-15T12:32:56.357 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-05-15T12:32:56.357 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-05-15T12:32:56.652 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-05-15T12:32:56.652 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:32:56.652 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-05-15T12:32:56.652 INFO:teuthology.orchestra.run.smithi178.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-15T12:32:56.652 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-grafana-dashboards-2:19.0.0-1882.gae248125.el9.noarch 2024-05-15T12:32:56.652 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-dashboard-2:19.0.0-1882.gae248125.el9.noarch 2024-05-15T12:32:56.652 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-prometheus-alerts-2:19.0.0-1882.gae248125.el9.noarch 2024-05-15T12:32:56.652 INFO:teuthology.orchestra.run.smithi178.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-15T12:32:56.652 INFO:teuthology.orchestra.run.smithi178.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-15T12:32:56.652 INFO:teuthology.orchestra.run.smithi178.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-15T12:32:56.652 INFO:teuthology.orchestra.run.smithi178.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-15T12:32:56.652 INFO:teuthology.orchestra.run.smithi178.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-15T12:32:56.653 INFO:teuthology.orchestra.run.smithi178.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-15T12:32:56.653 INFO:teuthology.orchestra.run.smithi178.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-15T12:32:56.653 INFO:teuthology.orchestra.run.smithi178.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-15T12:32:56.653 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:32:56.653 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-05-15T12:32:56.892 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-mgr-diskprediction-local 2024-05-15T12:32:56.930 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1882.gae248125.el9.x86_64 2/2 2024-05-15T12:32:56.931 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-fuse-2:19.0.0-1882.gae248125.el9.x86_64 1/2 2024-05-15T12:32:57.193 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-05-15T12:32:57.193 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:32:57.193 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-05-15T12:32:57.193 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-fuse-2:19.0.0-1882.gae248125.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-05-15T12:32:57.193 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:32:57.194 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-05-15T12:32:57.450 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-volume 2024-05-15T12:32:57.464 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:32 ago on Wed 15 May 2024 12:31:25 PM UTC. 2024-05-15T12:32:57.587 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-05-15T12:32:57.588 INFO:teuthology.orchestra.run.smithi178.stdout:======================================================================================= 2024-05-15T12:32:57.588 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-05-15T12:32:57.588 INFO:teuthology.orchestra.run.smithi178.stdout:======================================================================================= 2024-05-15T12:32:57.588 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-05-15T12:32:57.588 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1882.gae248125.el9 ceph-noarch 7.4 M 2024-05-15T12:32:57.588 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-05-15T12:32:57.588 INFO:teuthology.orchestra.run.smithi178.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-15T12:32:57.589 INFO:teuthology.orchestra.run.smithi178.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-15T12:32:57.589 INFO:teuthology.orchestra.run.smithi178.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-15T12:32:57.589 INFO:teuthology.orchestra.run.smithi178.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-15T12:32:57.589 INFO:teuthology.orchestra.run.smithi178.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-15T12:32:57.589 INFO:teuthology.orchestra.run.smithi178.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-15T12:32:57.589 INFO:teuthology.orchestra.run.smithi178.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-15T12:32:57.589 INFO:teuthology.orchestra.run.smithi178.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-15T12:32:57.589 INFO:teuthology.orchestra.run.smithi178.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-15T12:32:57.589 INFO:teuthology.orchestra.run.smithi178.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-15T12:32:57.589 INFO:teuthology.orchestra.run.smithi178.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-15T12:32:57.589 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:32:57.589 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-05-15T12:32:57.589 INFO:teuthology.orchestra.run.smithi178.stdout:======================================================================================= 2024-05-15T12:32:57.589 INFO:teuthology.orchestra.run.smithi178.stdout:Install 12 Packages 2024-05-15T12:32:57.589 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:32:57.590 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 38 M 2024-05-15T12:32:57.590 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 209 M 2024-05-15T12:32:57.590 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-05-15T12:32:58.038 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:35 ago on Wed 15 May 2024 12:31:23 PM UTC. 2024-05-15T12:32:58.167 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-05-15T12:32:58.168 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:32:58.168 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-05-15T12:32:58.168 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:32:58.168 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-05-15T12:32:58.168 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-volume noarch 2:19.0.0-1882.gae248125.el9 ceph-noarch 264 k 2024-05-15T12:32:58.168 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-05-15T12:32:58.169 INFO:teuthology.orchestra.run.smithi117.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-15T12:32:58.169 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:32:58.169 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-05-15T12:32:58.169 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:32:58.169 INFO:teuthology.orchestra.run.smithi117.stdout:Install 2 Packages 2024-05-15T12:32:58.169 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:32:58.169 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 341 k 2024-05-15T12:32:58.169 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 1.5 M 2024-05-15T12:32:58.169 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-05-15T12:32:58.300 INFO:teuthology.orchestra.run.smithi178.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 370 kB/s | 195 kB 00:00 2024-05-15T12:32:58.316 INFO:teuthology.orchestra.run.smithi117.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 996 kB/s | 77 kB 00:00 2024-05-15T12:32:58.417 INFO:teuthology.orchestra.run.smithi178.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 281 kB/s | 33 kB 00:00 2024-05-15T12:32:58.442 INFO:teuthology.orchestra.run.smithi178.stdout:(3/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 803 kB 00:00 2024-05-15T12:32:58.535 INFO:teuthology.orchestra.run.smithi178.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 198 kB/s | 18 kB 00:00 2024-05-15T12:32:58.618 INFO:teuthology.orchestra.run.smithi178.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 423 kB/s | 35 kB 00:00 2024-05-15T12:32:58.625 INFO:teuthology.orchestra.run.smithi117.stdout:(2/2): ceph-volume-19.0.0-1882.gae248125.el9.no 683 kB/s | 264 kB 00:00 2024-05-15T12:32:58.625 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:32:58.626 INFO:teuthology.orchestra.run.smithi117.stdout:Total 744 kB/s | 341 kB 00:00 2024-05-15T12:32:58.638 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-05-15T12:32:58.647 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-05-15T12:32:58.648 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-05-15T12:32:58.678 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-05-15T12:32:58.678 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-05-15T12:32:58.817 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-05-15T12:32:58.993 INFO:teuthology.orchestra.run.smithi178.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.2 MB/s | 3.0 MB 00:00 2024-05-15T12:32:59.000 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-05-15T12:32:59.044 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-volume-2:19.0.0-1882.gae248125.el9.noarch 2/2 2024-05-15T12:32:59.061 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-volume-2:19.0.0-1882.gae248125.el9.noarch 2/2 2024-05-15T12:32:59.061 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-15T12:32:59.061 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-15T12:32:59.061 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:32:59.077 INFO:teuthology.orchestra.run.smithi178.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 2.9 MB/s | 244 kB 00:00 2024-05-15T12:32:59.711 INFO:teuthology.orchestra.run.smithi178.stdout:(8/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 4.3 MB/s | 4.7 MB 00:01 2024-05-15T12:32:59.819 INFO:teuthology.orchestra.run.smithi178.stdout:(9/12): ceph-mgr-diskprediction-local-19.0.0-18 3.6 MB/s | 7.4 MB 00:02 2024-05-15T12:32:59.829 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-volume-2:19.0.0-1882.gae248125.el9.noarch 1/2 2024-05-15T12:32:59.903 INFO:teuthology.orchestra.run.smithi178.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 1.3 MB/s | 250 kB 00:00 2024-05-15T12:33:00.100 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-05-15T12:33:00.100 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:33:00.100 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-05-15T12:33:00.100 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-volume-2:19.0.0-1882.gae248125.el9.noarch 2024-05-15T12:33:00.100 INFO:teuthology.orchestra.run.smithi117.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-15T12:33:00.101 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:33:00.101 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-05-15T12:33:00.380 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install librados-devel 2024-05-15T12:33:00.503 INFO:teuthology.orchestra.run.smithi178.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.6 MB/s | 5.1 MB 00:01 2024-05-15T12:33:00.968 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:37 ago on Wed 15 May 2024 12:31:23 PM UTC. 2024-05-15T12:33:01.093 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-05-15T12:33:01.094 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:33:01.094 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-05-15T12:33:01.095 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:33:01.095 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-05-15T12:33:01.095 INFO:teuthology.orchestra.run.smithi117.stdout: librados-devel x86_64 2:19.0.0-1882.gae248125.el9 ceph 127 k 2024-05-15T12:33:01.095 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:33:01.095 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-05-15T12:33:01.095 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:33:01.095 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-05-15T12:33:01.095 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:33:01.095 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 127 k 2024-05-15T12:33:01.095 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 456 k 2024-05-15T12:33:01.095 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-05-15T12:33:01.306 INFO:teuthology.orchestra.run.smithi117.stdout:librados-devel-19.0.0-1882.gae248125.el9.x86_64 603 kB/s | 127 kB 00:00 2024-05-15T12:33:01.307 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:33:01.307 INFO:teuthology.orchestra.run.smithi117.stdout:Total 598 kB/s | 127 kB 00:00 2024-05-15T12:33:01.307 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-05-15T12:33:01.312 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-05-15T12:33:01.312 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-05-15T12:33:01.376 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-05-15T12:33:01.376 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-05-15T12:33:01.529 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-05-15T12:33:01.654 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librados-devel-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:33:02.117 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librados-devel-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:33:02.363 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librados-devel-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:33:02.363 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:33:02.363 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-05-15T12:33:02.363 INFO:teuthology.orchestra.run.smithi117.stdout: librados-devel-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:33:02.363 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:33:02.363 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-05-15T12:33:02.636 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install libcephfs2 2024-05-15T12:33:03.218 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:40 ago on Wed 15 May 2024 12:31:23 PM UTC. 2024-05-15T12:33:03.301 INFO:teuthology.orchestra.run.smithi117.stdout:Package libcephfs2-2:19.0.0-1882.gae248125.el9.x86_64 is already installed. 2024-05-15T12:33:03.345 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-05-15T12:33:03.346 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-05-15T12:33:03.346 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-05-15T12:33:03.402 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install libcephfs-devel 2024-05-15T12:33:03.653 INFO:teuthology.orchestra.run.smithi178.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 4.2 MB/s | 16 MB 00:03 2024-05-15T12:33:03.654 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:33:03.654 INFO:teuthology.orchestra.run.smithi178.stdout:Total 6.3 MB/s | 38 MB 00:06 2024-05-15T12:33:03.981 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:40 ago on Wed 15 May 2024 12:31:23 PM UTC. 2024-05-15T12:33:04.007 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-05-15T12:33:04.033 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-05-15T12:33:04.033 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-05-15T12:33:04.103 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-05-15T12:33:04.104 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:33:04.104 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-05-15T12:33:04.104 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:33:04.104 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-05-15T12:33:04.104 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs-devel x86_64 2:19.0.0-1882.gae248125.el9 ceph 31 k 2024-05-15T12:33:04.104 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:33:04.104 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-05-15T12:33:04.104 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:33:04.104 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-05-15T12:33:04.105 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:33:04.105 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 31 k 2024-05-15T12:33:04.105 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 140 k 2024-05-15T12:33:04.105 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-05-15T12:33:04.267 INFO:teuthology.orchestra.run.smithi117.stdout:libcephfs-devel-19.0.0-1882.gae248125.el9.x86_6 192 kB/s | 31 kB 00:00 2024-05-15T12:33:04.268 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:33:04.268 INFO:teuthology.orchestra.run.smithi117.stdout:Total 190 kB/s | 31 kB 00:00 2024-05-15T12:33:04.269 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-05-15T12:33:04.272 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-05-15T12:33:04.272 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-05-15T12:33:04.283 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-05-15T12:33:04.283 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-05-15T12:33:04.364 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-05-15T12:33:04.466 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libcephfs-devel-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:33:04.467 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-05-15T12:33:04.468 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-05-15T12:33:04.826 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:33:05.057 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libcephfs-devel-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:33:05.057 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:33:05.057 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-05-15T12:33:05.057 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs-devel-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:33:05.057 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:33:05.057 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-05-15T12:33:05.215 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install librados2 2024-05-15T12:33:05.232 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-05-15T12:33:05.311 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-05-15T12:33:05.351 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-05-15T12:33:05.453 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-05-15T12:33:05.494 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-05-15T12:33:05.671 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-05-15T12:33:05.701 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-05-15T12:33:05.781 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-05-15T12:33:05.793 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:42 ago on Wed 15 May 2024 12:31:23 PM UTC. 2024-05-15T12:33:05.874 INFO:teuthology.orchestra.run.smithi117.stdout:Package librados2-2:19.0.0-1882.gae248125.el9.x86_64 is already installed. 2024-05-15T12:33:05.915 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-05-15T12:33:05.916 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-05-15T12:33:05.916 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-05-15T12:33:05.975 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install librbd1 2024-05-15T12:33:06.450 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-05-15T12:33:06.550 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:43 ago on Wed 15 May 2024 12:31:23 PM UTC. 2024-05-15T12:33:06.566 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-05-15T12:33:06.631 INFO:teuthology.orchestra.run.smithi117.stdout:Package librbd1-2:19.0.0-1882.gae248125.el9.x86_64 is already installed. 2024-05-15T12:33:06.672 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-05-15T12:33:06.673 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-05-15T12:33:06.673 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-05-15T12:33:06.731 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install python3-rados 2024-05-15T12:33:07.306 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:44 ago on Wed 15 May 2024 12:31:23 PM UTC. 2024-05-15T12:33:07.386 INFO:teuthology.orchestra.run.smithi117.stdout:Package python3-rados-2:19.0.0-1882.gae248125.el9.x86_64 is already installed. 2024-05-15T12:33:07.427 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-05-15T12:33:07.428 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-05-15T12:33:07.428 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-05-15T12:33:07.482 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install python3-rgw 2024-05-15T12:33:07.820 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-05-15T12:33:08.056 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:45 ago on Wed 15 May 2024 12:31:23 PM UTC. 2024-05-15T12:33:08.136 INFO:teuthology.orchestra.run.smithi117.stdout:Package python3-rgw-2:19.0.0-1882.gae248125.el9.x86_64 is already installed. 2024-05-15T12:33:08.178 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-05-15T12:33:08.179 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-05-15T12:33:08.179 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-05-15T12:33:08.241 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install python3-cephfs 2024-05-15T12:33:08.816 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:45 ago on Wed 15 May 2024 12:31:23 PM UTC. 2024-05-15T12:33:08.869 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-05-15T12:33:08.897 INFO:teuthology.orchestra.run.smithi117.stdout:Package python3-cephfs-2:19.0.0-1882.gae248125.el9.x86_64 is already installed. 2024-05-15T12:33:08.902 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1882.gae248 12/12 2024-05-15T12:33:08.938 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-05-15T12:33:08.939 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-05-15T12:33:08.939 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-05-15T12:33:08.998 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install python3-rbd 2024-05-15T12:33:09.576 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:46 ago on Wed 15 May 2024 12:31:23 PM UTC. 2024-05-15T12:33:09.657 INFO:teuthology.orchestra.run.smithi117.stdout:Package python3-rbd-2:19.0.0-1882.gae248125.el9.x86_64 is already installed. 2024-05-15T12:33:09.699 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-05-15T12:33:09.700 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-05-15T12:33:09.700 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-05-15T12:33:09.766 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install rbd-fuse 2024-05-15T12:33:10.340 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:47 ago on Wed 15 May 2024 12:31:23 PM UTC. 2024-05-15T12:33:10.462 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-05-15T12:33:10.462 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:33:10.462 INFO:teuthology.orchestra.run.smithi117.stdout: Package Architecture Version Repository Size 2024-05-15T12:33:10.462 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:33:10.462 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-05-15T12:33:10.462 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-fuse x86_64 2:19.0.0-1882.gae248125.el9 ceph 85 k 2024-05-15T12:33:10.462 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:33:10.462 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-05-15T12:33:10.462 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:33:10.462 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-05-15T12:33:10.463 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:33:10.463 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 85 k 2024-05-15T12:33:10.463 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 231 k 2024-05-15T12:33:10.463 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-05-15T12:33:10.770 INFO:teuthology.orchestra.run.smithi117.stdout:rbd-fuse-19.0.0-1882.gae248125.el9.x86_64.rpm 278 kB/s | 85 kB 00:00 2024-05-15T12:33:10.771 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:33:10.771 INFO:teuthology.orchestra.run.smithi117.stdout:Total 277 kB/s | 85 kB 00:00 2024-05-15T12:33:10.771 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-05-15T12:33:10.778 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-05-15T12:33:10.778 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-05-15T12:33:10.839 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-05-15T12:33:10.839 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-05-15T12:33:10.994 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-05-15T12:33:11.076 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : rbd-fuse-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:33:11.315 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1882.gae248 12/12 2024-05-15T12:33:11.315 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1882.gae248 1/12 2024-05-15T12:33:11.315 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-05-15T12:33:11.315 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-05-15T12:33:11.315 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-05-15T12:33:11.315 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-05-15T12:33:11.315 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-05-15T12:33:11.315 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-05-15T12:33:11.315 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-05-15T12:33:11.315 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-05-15T12:33:11.316 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-05-15T12:33:11.316 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-05-15T12:33:11.466 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:33:11.676 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-05-15T12:33:11.676 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:33:11.676 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-05-15T12:33:11.676 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1882.gae248125.el9.noarch 2024-05-15T12:33:11.677 INFO:teuthology.orchestra.run.smithi178.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-15T12:33:11.677 INFO:teuthology.orchestra.run.smithi178.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-15T12:33:11.677 INFO:teuthology.orchestra.run.smithi178.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-15T12:33:11.677 INFO:teuthology.orchestra.run.smithi178.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-15T12:33:11.677 INFO:teuthology.orchestra.run.smithi178.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-15T12:33:11.677 INFO:teuthology.orchestra.run.smithi178.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-15T12:33:11.678 INFO:teuthology.orchestra.run.smithi178.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-15T12:33:11.678 INFO:teuthology.orchestra.run.smithi178.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-15T12:33:11.678 INFO:teuthology.orchestra.run.smithi178.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-15T12:33:11.678 INFO:teuthology.orchestra.run.smithi178.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-15T12:33:11.678 INFO:teuthology.orchestra.run.smithi178.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-15T12:33:11.678 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:33:11.679 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-05-15T12:33:11.714 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : rbd-fuse-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:33:11.714 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:33:11.715 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-05-15T12:33:11.715 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-fuse-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:33:11.715 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:33:11.715 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-05-15T12:33:11.933 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install rbd-mirror 2024-05-15T12:33:12.064 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-mgr-rook 2024-05-15T12:33:12.539 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:49 ago on Wed 15 May 2024 12:31:23 PM UTC. 2024-05-15T12:33:12.651 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:47 ago on Wed 15 May 2024 12:31:25 PM UTC. 2024-05-15T12:33:12.667 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-05-15T12:33:12.668 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:33:12.669 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-05-15T12:33:12.669 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:33:12.669 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-05-15T12:33:12.669 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-mirror x86_64 2:19.0.0-1882.gae248125.el9 ceph 3.1 M 2024-05-15T12:33:12.669 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:33:12.669 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-05-15T12:33:12.670 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:33:12.670 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-05-15T12:33:12.670 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:33:12.670 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 3.1 M 2024-05-15T12:33:12.670 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 13 M 2024-05-15T12:33:12.671 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-05-15T12:33:12.775 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-05-15T12:33:12.776 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:33:12.776 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-05-15T12:33:12.776 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:33:12.776 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-05-15T12:33:12.776 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-rook noarch 2:19.0.0-1882.gae248125.el9 ceph-noarch 49 k 2024-05-15T12:33:12.777 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-05-15T12:33:12.777 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-15T12:33:12.777 INFO:teuthology.orchestra.run.smithi178.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-15T12:33:12.777 INFO:teuthology.orchestra.run.smithi178.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-15T12:33:12.777 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-15T12:33:12.777 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-15T12:33:12.777 INFO:teuthology.orchestra.run.smithi178.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-15T12:33:12.777 INFO:teuthology.orchestra.run.smithi178.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-15T12:33:12.777 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-15T12:33:12.777 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-15T12:33:12.777 INFO:teuthology.orchestra.run.smithi178.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-15T12:33:12.777 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-15T12:33:12.777 INFO:teuthology.orchestra.run.smithi178.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-15T12:33:12.777 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:33:12.777 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-05-15T12:33:12.777 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:33:12.778 INFO:teuthology.orchestra.run.smithi178.stdout:Install 13 Packages 2024-05-15T12:33:12.778 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:33:12.778 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 2.2 M 2024-05-15T12:33:12.778 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 26 M 2024-05-15T12:33:12.778 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-05-15T12:33:13.224 INFO:teuthology.orchestra.run.smithi117.stdout:rbd-mirror-19.0.0-1882.gae248125.el9.x86_64.rpm 5.6 MB/s | 3.1 MB 00:00 2024-05-15T12:33:13.225 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:33:13.225 INFO:teuthology.orchestra.run.smithi117.stdout:Total 5.5 MB/s | 3.1 MB 00:00 2024-05-15T12:33:13.226 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-05-15T12:33:13.235 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-05-15T12:33:13.235 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-05-15T12:33:13.317 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-05-15T12:33:13.318 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-05-15T12:33:13.354 INFO:teuthology.orchestra.run.smithi178.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 412 kB/s | 26 kB 00:00 2024-05-15T12:33:13.379 INFO:teuthology.orchestra.run.smithi178.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 220 kB/s | 19 kB 00:00 2024-05-15T12:33:13.487 INFO:teuthology.orchestra.run.smithi178.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 222 kB 00:00 2024-05-15T12:33:13.513 INFO:teuthology.orchestra.run.smithi178.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-05-15T12:33:13.579 INFO:teuthology.orchestra.run.smithi178.stdout:(5/13): python3-requests-oauthlib-1.3.0-12.el9. 806 kB/s | 54 kB 00:00 2024-05-15T12:33:13.605 INFO:teuthology.orchestra.run.smithi178.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.3 MB/s | 279 kB 00:00 2024-05-15T12:33:13.750 INFO:teuthology.orchestra.run.smithi178.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 106 kB/s | 15 kB 00:00 2024-05-15T12:33:13.758 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-05-15T12:33:13.772 INFO:teuthology.orchestra.run.smithi178.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 167 kB/s | 32 kB 00:00 2024-05-15T12:33:13.798 INFO:teuthology.orchestra.run.smithi178.stdout:(9/13): ceph-mgr-rook-19.0.0-1882.gae248125.el9 97 kB/s | 49 kB 00:00 2024-05-15T12:33:13.816 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : rbd-mirror-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:33:13.843 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:33:13.843 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-15T12:33:13.843 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-15T12:33:13.843 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-15T12:33:13.843 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-15T12:33:13.843 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:33:13.869 INFO:teuthology.orchestra.run.smithi178.stdout:(10/13): python3-google-auth-2.29.0-1.el9.noarc 1.7 MB/s | 206 kB 00:00 2024-05-15T12:33:13.894 INFO:teuthology.orchestra.run.smithi178.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 616 kB/s | 59 kB 00:00 2024-05-15T12:33:13.919 INFO:teuthology.orchestra.run.smithi178.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-05-15T12:33:13.953 INFO:teuthology.orchestra.run.smithi178.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 5.7 MB/s | 1.0 MB 00:00 2024-05-15T12:33:13.953 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:33:13.954 INFO:teuthology.orchestra.run.smithi178.stdout:Total 1.9 MB/s | 2.2 MB 00:01 2024-05-15T12:33:14.019 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-05-15T12:33:14.033 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-05-15T12:33:14.033 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-05-15T12:33:14.118 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-05-15T12:33:14.118 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-05-15T12:33:14.634 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-05-15T12:33:14.800 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-05-15T12:33:14.870 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-05-15T12:33:14.915 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : rbd-mirror-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:33:14.915 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:33:14.915 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-05-15T12:33:14.915 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-mirror-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:33:14.915 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:33:14.915 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-05-15T12:33:14.925 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-05-15T12:33:14.963 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-05-15T12:33:15.004 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-05-15T12:33:15.098 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-05-15T12:33:15.134 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install rbd-nbd 2024-05-15T12:33:15.208 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 7/13 2024-05-15T12:33:15.256 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-05-15T12:33:15.721 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:52 ago on Wed 15 May 2024 12:31:23 PM UTC. 2024-05-15T12:33:15.846 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-05-15T12:33:15.847 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:33:15.847 INFO:teuthology.orchestra.run.smithi117.stdout: Package Architecture Version Repository Size 2024-05-15T12:33:15.847 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:33:15.847 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-05-15T12:33:15.847 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-nbd x86_64 2:19.0.0-1882.gae248125.el9 ceph 171 k 2024-05-15T12:33:15.847 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:33:15.847 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-05-15T12:33:15.847 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:33:15.847 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-05-15T12:33:15.848 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:33:15.848 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 171 k 2024-05-15T12:33:15.848 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 490 k 2024-05-15T12:33:15.848 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-05-15T12:33:15.974 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-05-15T12:33:16.028 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-05-15T12:33:16.061 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-05-15T12:33:16.109 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-05-15T12:33:16.132 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-rook-2:19.0.0-1882.gae248125.el9.noarch 13/13 2024-05-15T12:33:16.198 INFO:teuthology.orchestra.run.smithi117.stdout:rbd-nbd-19.0.0-1882.gae248125.el9.x86_64.rpm 487 kB/s | 171 kB 00:00 2024-05-15T12:33:16.199 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:33:16.199 INFO:teuthology.orchestra.run.smithi117.stdout:Total 485 kB/s | 171 kB 00:00 2024-05-15T12:33:16.199 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-05-15T12:33:16.207 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-05-15T12:33:16.207 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-05-15T12:33:16.270 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-05-15T12:33:16.270 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-05-15T12:33:16.427 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-05-15T12:33:16.538 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : rbd-nbd-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:33:16.922 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1882.gae248125.el9.noarch 13/13 2024-05-15T12:33:16.923 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1882.gae248125.el9.noarch 1/13 2024-05-15T12:33:16.923 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-05-15T12:33:16.923 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-05-15T12:33:16.923 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-05-15T12:33:16.923 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-05-15T12:33:16.923 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-05-15T12:33:16.923 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-05-15T12:33:16.923 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-05-15T12:33:16.923 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-05-15T12:33:16.923 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 10/13 2024-05-15T12:33:16.923 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-05-15T12:33:16.923 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-05-15T12:33:16.938 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:33:17.188 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : rbd-nbd-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:33:17.188 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:33:17.188 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-05-15T12:33:17.188 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-nbd-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:33:17.188 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:33:17.188 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-05-15T12:33:17.241 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-05-15T12:33:17.242 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:33:17.242 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-05-15T12:33:17.242 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-rook-2:19.0.0-1882.gae248125.el9.noarch 2024-05-15T12:33:17.242 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-15T12:33:17.242 INFO:teuthology.orchestra.run.smithi178.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-15T12:33:17.242 INFO:teuthology.orchestra.run.smithi178.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-15T12:33:17.242 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-15T12:33:17.242 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-15T12:33:17.242 INFO:teuthology.orchestra.run.smithi178.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-15T12:33:17.242 INFO:teuthology.orchestra.run.smithi178.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-15T12:33:17.242 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-15T12:33:17.242 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-15T12:33:17.243 INFO:teuthology.orchestra.run.smithi178.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-15T12:33:17.243 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-15T12:33:17.243 INFO:teuthology.orchestra.run.smithi178.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-15T12:33:17.243 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:33:17.243 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-05-15T12:33:17.410 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-base-debuginfo 2024-05-15T12:33:17.486 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-mgr-cephadm 2024-05-15T12:33:17.990 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:55 ago on Wed 15 May 2024 12:31:23 PM UTC. 2024-05-15T12:33:18.071 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:53 ago on Wed 15 May 2024 12:31:25 PM UTC. 2024-05-15T12:33:18.113 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-05-15T12:33:18.114 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:33:18.115 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-05-15T12:33:18.115 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:33:18.115 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-05-15T12:33:18.115 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 68 M 2024-05-15T12:33:18.115 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-05-15T12:33:18.115 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 23 M 2024-05-15T12:33:18.115 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:33:18.115 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-05-15T12:33:18.115 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:33:18.115 INFO:teuthology.orchestra.run.smithi117.stdout:Install 2 Packages 2024-05-15T12:33:18.115 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:33:18.115 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 91 M 2024-05-15T12:33:18.115 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 501 M 2024-05-15T12:33:18.115 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-05-15T12:33:18.194 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-05-15T12:33:18.195 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:33:18.195 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-05-15T12:33:18.195 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:33:18.195 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-05-15T12:33:18.195 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1882.gae248125.el9 ceph-noarch 138 k 2024-05-15T12:33:18.195 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-05-15T12:33:18.195 INFO:teuthology.orchestra.run.smithi178.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-15T12:33:18.195 INFO:teuthology.orchestra.run.smithi178.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-15T12:33:18.195 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-05-15T12:33:18.195 INFO:teuthology.orchestra.run.smithi178.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-15T12:33:18.195 INFO:teuthology.orchestra.run.smithi178.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-15T12:33:18.196 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:33:18.196 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-05-15T12:33:18.196 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:33:18.196 INFO:teuthology.orchestra.run.smithi178.stdout:Install 6 Packages 2024-05-15T12:33:18.196 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:33:18.196 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 7.0 M 2024-05-15T12:33:18.196 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 33 M 2024-05-15T12:33:18.196 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-05-15T12:33:18.771 INFO:teuthology.orchestra.run.smithi178.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.7 MB/s | 248 kB 00:00 2024-05-15T12:33:18.938 INFO:teuthology.orchestra.run.smithi178.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 6.0 MB 00:00 2024-05-15T12:33:18.980 INFO:teuthology.orchestra.run.smithi178.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.6 MB/s | 548 kB 00:00 2024-05-15T12:33:19.022 INFO:teuthology.orchestra.run.smithi178.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 1.2 MB/s | 51 kB 00:00 2024-05-15T12:33:19.047 INFO:teuthology.orchestra.run.smithi178.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 531 kB/s | 58 kB 00:00 2024-05-15T12:33:19.156 INFO:teuthology.orchestra.run.smithi178.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1882.gae248125.e 260 kB/s | 138 kB 00:00 2024-05-15T12:33:19.156 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:33:19.157 INFO:teuthology.orchestra.run.smithi178.stdout:Total 7.3 MB/s | 7.0 MB 00:00 2024-05-15T12:33:19.228 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-05-15T12:33:19.239 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-05-15T12:33:19.240 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-05-15T12:33:19.337 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-05-15T12:33:19.337 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-05-15T12:33:19.638 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-05-15T12:33:19.813 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-05-15T12:33:19.866 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-05-15T12:33:20.214 INFO:teuthology.orchestra.run.smithi117.stdout:(1/2): ceph-debuginfo-19.0.0-1882.gae248125.el9 11 MB/s | 23 MB 00:02 2024-05-15T12:33:20.385 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-05-15T12:33:20.477 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-05-15T12:33:20.538 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-05-15T12:33:20.561 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1882.gae248125.el9.noarch 6/6 2024-05-15T12:33:21.384 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1882.gae248125.el9.noarch 6/6 2024-05-15T12:33:21.384 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1882.gae248125.el9.noarch 1/6 2024-05-15T12:33:21.384 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-05-15T12:33:21.384 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-05-15T12:33:21.384 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-05-15T12:33:21.384 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-05-15T12:33:21.654 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-05-15T12:33:21.655 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:33:21.655 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-05-15T12:33:21.655 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-cephadm-2:19.0.0-1882.gae248125.el9.noarch 2024-05-15T12:33:21.655 INFO:teuthology.orchestra.run.smithi178.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-15T12:33:21.655 INFO:teuthology.orchestra.run.smithi178.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-15T12:33:21.655 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-05-15T12:33:21.655 INFO:teuthology.orchestra.run.smithi178.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-15T12:33:21.655 INFO:teuthology.orchestra.run.smithi178.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-15T12:33:21.655 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:33:21.655 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-05-15T12:33:21.921 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-fuse 2024-05-15T12:33:22.313 INFO:teuthology.orchestra.run.smithi117.stdout:(2/2): ceph-base-debuginfo-19.0.0-1882.gae24812 16 MB/s | 68 MB 00:04 2024-05-15T12:33:22.315 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:33:22.316 INFO:teuthology.orchestra.run.smithi117.stdout:Total 22 MB/s | 91 MB 00:04 2024-05-15T12:33:22.316 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-05-15T12:33:22.320 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-05-15T12:33:22.320 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-05-15T12:33:22.502 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:57 ago on Wed 15 May 2024 12:31:25 PM UTC. 2024-05-15T12:33:22.623 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-05-15T12:33:22.624 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:33:22.624 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-05-15T12:33:22.624 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:33:22.624 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-05-15T12:33:22.624 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-fuse x86_64 2:19.0.0-1882.gae248125.el9 ceph 888 k 2024-05-15T12:33:22.624 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-05-15T12:33:22.624 INFO:teuthology.orchestra.run.smithi178.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-15T12:33:22.624 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:33:22.624 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-05-15T12:33:22.624 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:33:22.625 INFO:teuthology.orchestra.run.smithi178.stdout:Install 2 Packages 2024-05-15T12:33:22.625 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:33:22.625 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 968 k 2024-05-15T12:33:22.625 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 2.8 M 2024-05-15T12:33:22.626 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-05-15T12:33:22.866 INFO:teuthology.orchestra.run.smithi178.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 664 kB/s | 80 kB 00:00 2024-05-15T12:33:22.907 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-05-15T12:33:22.907 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-05-15T12:33:23.191 INFO:teuthology.orchestra.run.smithi178.stdout:(2/2): ceph-fuse-19.0.0-1882.gae248125.el9.x86_ 1.9 MB/s | 888 kB 00:00 2024-05-15T12:33:23.192 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:33:23.192 INFO:teuthology.orchestra.run.smithi178.stdout:Total 1.7 MB/s | 968 kB 00:00 2024-05-15T12:33:23.204 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-05-15T12:33:23.226 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-05-15T12:33:23.226 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-05-15T12:33:23.305 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-05-15T12:33:23.306 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-05-15T12:33:23.494 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-05-15T12:33:23.627 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-05-15T12:33:23.707 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-fuse-2:19.0.0-1882.gae248125.el9.x86_64 2/2 2024-05-15T12:33:24.333 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1882.gae248125.el9.x86_64 2/2 2024-05-15T12:33:24.334 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-fuse-2:19.0.0-1882.gae248125.el9.x86_64 1/2 2024-05-15T12:33:24.588 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-05-15T12:33:24.589 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:33:24.589 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-05-15T12:33:24.589 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-fuse-2:19.0.0-1882.gae248125.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-05-15T12:33:24.589 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:33:24.589 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-05-15T12:33:24.853 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-volume 2024-05-15T12:33:25.440 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:02:00 ago on Wed 15 May 2024 12:31:25 PM UTC. 2024-05-15T12:33:25.571 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-05-15T12:33:25.571 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:33:25.571 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-05-15T12:33:25.571 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:33:25.572 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-05-15T12:33:25.572 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-volume noarch 2:19.0.0-1882.gae248125.el9 ceph-noarch 264 k 2024-05-15T12:33:25.572 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-05-15T12:33:25.572 INFO:teuthology.orchestra.run.smithi178.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-15T12:33:25.572 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:33:25.572 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-05-15T12:33:25.572 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:33:25.572 INFO:teuthology.orchestra.run.smithi178.stdout:Install 2 Packages 2024-05-15T12:33:25.572 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:33:25.572 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 341 k 2024-05-15T12:33:25.572 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 1.5 M 2024-05-15T12:33:25.572 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-05-15T12:33:25.771 INFO:teuthology.orchestra.run.smithi178.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 481 kB/s | 77 kB 00:00 2024-05-15T12:33:25.854 INFO:teuthology.orchestra.run.smithi178.stdout:(2/2): ceph-volume-19.0.0-1882.gae248125.el9.no 1.1 MB/s | 264 kB 00:00 2024-05-15T12:33:25.854 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:33:25.855 INFO:teuthology.orchestra.run.smithi178.stdout:Total 1.2 MB/s | 341 kB 00:00 2024-05-15T12:33:25.867 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-05-15T12:33:25.877 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-05-15T12:33:25.877 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-05-15T12:33:25.910 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-05-15T12:33:25.911 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-05-15T12:33:26.056 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-05-15T12:33:26.059 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-05-15T12:33:26.230 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-05-15T12:33:26.273 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-volume-2:19.0.0-1882.gae248125.el9.noarch 2/2 2024-05-15T12:33:26.290 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-volume-2:19.0.0-1882.gae248125.el9.noarch 2/2 2024-05-15T12:33:26.290 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-15T12:33:26.290 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-15T12:33:26.290 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:33:27.008 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-volume-2:19.0.0-1882.gae248125.el9.noarch 1/2 2024-05-15T12:33:27.350 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-05-15T12:33:27.350 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:33:27.350 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-05-15T12:33:27.350 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-volume-2:19.0.0-1882.gae248125.el9.noarch 2024-05-15T12:33:27.350 INFO:teuthology.orchestra.run.smithi178.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-15T12:33:27.350 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:33:27.350 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-05-15T12:33:27.651 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install librados-devel 2024-05-15T12:33:28.250 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:02:03 ago on Wed 15 May 2024 12:31:25 PM UTC. 2024-05-15T12:33:28.374 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-05-15T12:33:28.374 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:33:28.374 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-05-15T12:33:28.375 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:33:28.375 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-05-15T12:33:28.375 INFO:teuthology.orchestra.run.smithi178.stdout: librados-devel x86_64 2:19.0.0-1882.gae248125.el9 ceph 127 k 2024-05-15T12:33:28.375 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:33:28.375 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-05-15T12:33:28.375 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:33:28.375 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-05-15T12:33:28.375 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:33:28.375 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 127 k 2024-05-15T12:33:28.375 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 456 k 2024-05-15T12:33:28.375 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-05-15T12:33:28.768 INFO:teuthology.orchestra.run.smithi178.stdout:librados-devel-19.0.0-1882.gae248125.el9.x86_64 323 kB/s | 127 kB 00:00 2024-05-15T12:33:28.768 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:33:28.769 INFO:teuthology.orchestra.run.smithi178.stdout:Total 322 kB/s | 127 kB 00:00 2024-05-15T12:33:28.769 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-05-15T12:33:28.774 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-05-15T12:33:28.774 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-05-15T12:33:28.839 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-05-15T12:33:28.839 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-05-15T12:33:28.990 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-05-15T12:33:29.082 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librados-devel-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:33:29.514 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librados-devel-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:33:29.744 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librados-devel-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:33:29.745 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:33:29.745 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-05-15T12:33:29.745 INFO:teuthology.orchestra.run.smithi178.stdout: librados-devel-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:33:29.745 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:33:29.745 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-05-15T12:33:30.012 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install libcephfs2 2024-05-15T12:33:30.595 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:02:05 ago on Wed 15 May 2024 12:31:25 PM UTC. 2024-05-15T12:33:30.676 INFO:teuthology.orchestra.run.smithi178.stdout:Package libcephfs2-2:19.0.0-1882.gae248125.el9.x86_64 is already installed. 2024-05-15T12:33:30.719 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-05-15T12:33:30.721 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-05-15T12:33:30.721 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-05-15T12:33:30.785 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install libcephfs-devel 2024-05-15T12:33:31.360 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:02:06 ago on Wed 15 May 2024 12:31:25 PM UTC. 2024-05-15T12:33:31.482 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-05-15T12:33:31.483 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:33:31.483 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-05-15T12:33:31.483 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:33:31.483 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-05-15T12:33:31.483 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs-devel x86_64 2:19.0.0-1882.gae248125.el9 ceph 31 k 2024-05-15T12:33:31.483 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:33:31.483 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-05-15T12:33:31.484 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:33:31.484 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-05-15T12:33:31.484 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:33:31.484 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 31 k 2024-05-15T12:33:31.484 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 140 k 2024-05-15T12:33:31.484 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-05-15T12:33:31.871 INFO:teuthology.orchestra.run.smithi178.stdout:libcephfs-devel-19.0.0-1882.gae248125.el9.x86_6 80 kB/s | 31 kB 00:00 2024-05-15T12:33:31.872 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:33:31.872 INFO:teuthology.orchestra.run.smithi178.stdout:Total 80 kB/s | 31 kB 00:00 2024-05-15T12:33:31.872 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-05-15T12:33:31.875 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-05-15T12:33:31.875 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-05-15T12:33:31.887 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-05-15T12:33:31.887 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-05-15T12:33:31.971 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-05-15T12:33:32.094 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libcephfs-devel-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:33:32.450 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:33:32.669 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libcephfs-devel-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:33:32.669 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:33:32.670 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-05-15T12:33:32.670 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs-devel-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:33:32.670 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:33:32.670 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-05-15T12:33:32.855 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install librados2 2024-05-15T12:33:33.209 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 1/2 2024-05-15T12:33:33.280 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1882.gae248125.el9.x86_ 2/2 2024-05-15T12:33:33.431 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:02:08 ago on Wed 15 May 2024 12:31:25 PM UTC. 2024-05-15T12:33:33.511 INFO:teuthology.orchestra.run.smithi178.stdout:Package librados2-2:19.0.0-1882.gae248125.el9.x86_64 is already installed. 2024-05-15T12:33:33.552 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-05-15T12:33:33.553 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-05-15T12:33:33.553 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-05-15T12:33:33.618 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install librbd1 2024-05-15T12:33:34.192 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:02:09 ago on Wed 15 May 2024 12:31:25 PM UTC. 2024-05-15T12:33:34.272 INFO:teuthology.orchestra.run.smithi178.stdout:Package librbd1-2:19.0.0-1882.gae248125.el9.x86_64 is already installed. 2024-05-15T12:33:34.316 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-05-15T12:33:34.317 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-05-15T12:33:34.317 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-05-15T12:33:34.383 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install python3-rados 2024-05-15T12:33:34.956 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:02:09 ago on Wed 15 May 2024 12:31:25 PM UTC. 2024-05-15T12:33:35.036 INFO:teuthology.orchestra.run.smithi178.stdout:Package python3-rados-2:19.0.0-1882.gae248125.el9.x86_64 is already installed. 2024-05-15T12:33:35.078 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-05-15T12:33:35.079 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-05-15T12:33:35.079 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-05-15T12:33:35.141 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install python3-rgw 2024-05-15T12:33:35.714 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:02:10 ago on Wed 15 May 2024 12:31:25 PM UTC. 2024-05-15T12:33:35.795 INFO:teuthology.orchestra.run.smithi178.stdout:Package python3-rgw-2:19.0.0-1882.gae248125.el9.x86_64 is already installed. 2024-05-15T12:33:35.836 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-05-15T12:33:35.837 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-05-15T12:33:35.838 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-05-15T12:33:35.901 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install python3-cephfs 2024-05-15T12:33:36.478 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:02:11 ago on Wed 15 May 2024 12:31:25 PM UTC. 2024-05-15T12:33:36.559 INFO:teuthology.orchestra.run.smithi178.stdout:Package python3-cephfs-2:19.0.0-1882.gae248125.el9.x86_64 is already installed. 2024-05-15T12:33:36.571 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1882.gae248125.el9.x86_ 2/2 2024-05-15T12:33:36.571 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1882.gae248125.el9.x86_ 1/2 2024-05-15T12:33:36.600 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-05-15T12:33:36.601 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-05-15T12:33:36.601 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-05-15T12:33:36.668 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install python3-rbd 2024-05-15T12:33:36.790 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2/2 2024-05-15T12:33:36.790 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:33:36.790 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-05-15T12:33:36.790 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-base-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:33:36.790 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:33:36.791 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:33:36.791 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-05-15T12:33:37.007 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-common-debuginfo 2024-05-15T12:33:37.244 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:02:12 ago on Wed 15 May 2024 12:31:25 PM UTC. 2024-05-15T12:33:37.325 INFO:teuthology.orchestra.run.smithi178.stdout:Package python3-rbd-2:19.0.0-1882.gae248125.el9.x86_64 is already installed. 2024-05-15T12:33:37.367 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-05-15T12:33:37.368 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-05-15T12:33:37.368 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-05-15T12:33:37.433 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install rbd-fuse 2024-05-15T12:33:37.585 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:02:14 ago on Wed 15 May 2024 12:31:23 PM UTC. 2024-05-15T12:33:37.708 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-05-15T12:33:37.709 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:33:37.709 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-05-15T12:33:37.709 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:33:37.709 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-05-15T12:33:37.709 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 422 M 2024-05-15T12:33:37.709 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:33:37.709 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-05-15T12:33:37.709 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:33:37.709 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-05-15T12:33:37.709 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:33:37.709 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 422 M 2024-05-15T12:33:37.709 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 1.7 G 2024-05-15T12:33:37.709 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-05-15T12:33:38.010 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:02:13 ago on Wed 15 May 2024 12:31:25 PM UTC. 2024-05-15T12:33:38.132 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-05-15T12:33:38.133 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:33:38.133 INFO:teuthology.orchestra.run.smithi178.stdout: Package Architecture Version Repository Size 2024-05-15T12:33:38.133 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:33:38.133 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-05-15T12:33:38.133 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-fuse x86_64 2:19.0.0-1882.gae248125.el9 ceph 85 k 2024-05-15T12:33:38.133 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:33:38.133 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-05-15T12:33:38.133 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:33:38.134 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-05-15T12:33:38.134 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:33:38.134 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 85 k 2024-05-15T12:33:38.134 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 231 k 2024-05-15T12:33:38.134 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-05-15T12:33:38.507 INFO:teuthology.orchestra.run.smithi178.stdout:rbd-fuse-19.0.0-1882.gae248125.el9.x86_64.rpm 229 kB/s | 85 kB 00:00 2024-05-15T12:33:38.508 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:33:38.508 INFO:teuthology.orchestra.run.smithi178.stdout:Total 228 kB/s | 85 kB 00:00 2024-05-15T12:33:38.508 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-05-15T12:33:38.515 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-05-15T12:33:38.515 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-05-15T12:33:38.577 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-05-15T12:33:38.578 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-05-15T12:33:38.724 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-05-15T12:33:38.822 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : rbd-fuse-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:33:39.194 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:33:39.451 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : rbd-fuse-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:33:39.451 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:33:39.451 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-05-15T12:33:39.451 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-fuse-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:33:39.451 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:33:39.451 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-05-15T12:33:39.698 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install rbd-mirror 2024-05-15T12:33:40.286 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:02:15 ago on Wed 15 May 2024 12:31:25 PM UTC. 2024-05-15T12:33:40.412 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-05-15T12:33:40.413 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:33:40.413 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-05-15T12:33:40.413 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:33:40.413 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-05-15T12:33:40.413 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-mirror x86_64 2:19.0.0-1882.gae248125.el9 ceph 3.1 M 2024-05-15T12:33:40.413 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:33:40.413 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-05-15T12:33:40.413 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:33:40.413 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-05-15T12:33:40.413 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:33:40.414 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 3.1 M 2024-05-15T12:33:40.414 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 13 M 2024-05-15T12:33:40.414 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-05-15T12:33:43.318 INFO:teuthology.orchestra.run.smithi178.stdout:rbd-mirror-19.0.0-1882.gae248125.el9.x86_64.rpm 1.1 MB/s | 3.1 MB 00:02 2024-05-15T12:33:43.319 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:33:43.320 INFO:teuthology.orchestra.run.smithi178.stdout:Total 1.1 MB/s | 3.1 MB 00:02 2024-05-15T12:33:43.320 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-05-15T12:33:43.330 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-05-15T12:33:43.330 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-05-15T12:33:43.415 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-05-15T12:33:43.415 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-05-15T12:33:43.848 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-05-15T12:33:43.919 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : rbd-mirror-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:33:43.944 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:33:43.945 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-15T12:33:43.945 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-15T12:33:43.945 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-15T12:33:43.945 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-15T12:33:43.945 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:33:44.933 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : rbd-mirror-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:33:44.933 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:33:44.933 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-05-15T12:33:44.933 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-mirror-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:33:44.933 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:33:44.933 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-05-15T12:33:45.182 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install rbd-nbd 2024-05-15T12:33:45.776 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:02:20 ago on Wed 15 May 2024 12:31:25 PM UTC. 2024-05-15T12:33:45.902 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-05-15T12:33:45.903 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:33:45.903 INFO:teuthology.orchestra.run.smithi178.stdout: Package Architecture Version Repository Size 2024-05-15T12:33:45.903 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:33:45.903 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-05-15T12:33:45.903 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-nbd x86_64 2:19.0.0-1882.gae248125.el9 ceph 171 k 2024-05-15T12:33:45.903 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:33:45.903 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-05-15T12:33:45.903 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:33:45.903 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-05-15T12:33:45.903 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:33:45.904 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 171 k 2024-05-15T12:33:45.904 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 490 k 2024-05-15T12:33:45.904 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-05-15T12:33:46.576 INFO:teuthology.orchestra.run.smithi178.stdout:rbd-nbd-19.0.0-1882.gae248125.el9.x86_64.rpm 254 kB/s | 171 kB 00:00 2024-05-15T12:33:46.576 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:33:46.576 INFO:teuthology.orchestra.run.smithi178.stdout:Total 254 kB/s | 171 kB 00:00 2024-05-15T12:33:46.577 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-05-15T12:33:46.584 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-05-15T12:33:46.584 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-05-15T12:33:46.648 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-05-15T12:33:46.648 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-05-15T12:33:46.799 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-05-15T12:33:46.906 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : rbd-nbd-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:33:47.315 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:33:47.562 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : rbd-nbd-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:33:47.562 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:33:47.562 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-05-15T12:33:47.562 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-nbd-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:33:47.562 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:33:47.562 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-05-15T12:33:47.838 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-base-debuginfo 2024-05-15T12:33:48.433 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:02:23 ago on Wed 15 May 2024 12:31:25 PM UTC. 2024-05-15T12:33:48.563 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-05-15T12:33:48.563 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:33:48.564 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-05-15T12:33:48.564 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:33:48.564 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-05-15T12:33:48.564 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 68 M 2024-05-15T12:33:48.564 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-05-15T12:33:48.564 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 23 M 2024-05-15T12:33:48.564 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:33:48.564 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-05-15T12:33:48.564 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:33:48.564 INFO:teuthology.orchestra.run.smithi178.stdout:Install 2 Packages 2024-05-15T12:33:48.565 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:33:48.565 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 91 M 2024-05-15T12:33:48.565 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 501 M 2024-05-15T12:33:48.565 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-05-15T12:33:56.565 INFO:teuthology.orchestra.run.smithi178.stdout:(1/2): ceph-debuginfo-19.0.0-1882.gae248125.el9 2.9 MB/s | 23 MB 00:07 2024-05-15T12:34:05.380 INFO:teuthology.orchestra.run.smithi178.stdout:(2/2): ceph-base-debuginfo-19.0.0-1882.gae24812 4.1 MB/s | 68 MB 00:16 2024-05-15T12:34:05.382 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:34:05.383 INFO:teuthology.orchestra.run.smithi178.stdout:Total 5.4 MB/s | 91 MB 00:16 2024-05-15T12:34:05.383 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-05-15T12:34:05.388 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-05-15T12:34:05.388 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-05-15T12:34:05.982 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-05-15T12:34:05.982 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-05-15T12:34:09.166 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-05-15T12:34:15.096 INFO:teuthology.orchestra.run.smithi117.stdout:ceph-common-debuginfo-19.0.0-1882.gae248125.el9 11 MB/s | 422 MB 00:37 2024-05-15T12:34:15.096 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:34:15.097 INFO:teuthology.orchestra.run.smithi117.stdout:Total 11 MB/s | 422 MB 00:37 2024-05-15T12:34:15.097 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-05-15T12:34:15.101 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-05-15T12:34:15.101 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-05-15T12:34:16.391 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 1/2 2024-05-15T12:34:16.465 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1882.gae248125.el9.x86_ 2/2 2024-05-15T12:34:17.604 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-05-15T12:34:17.605 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-05-15T12:34:19.929 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1882.gae248125.el9.x86_ 2/2 2024-05-15T12:34:19.929 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1882.gae248125.el9.x86_ 1/2 2024-05-15T12:34:20.159 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2/2 2024-05-15T12:34:20.159 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:34:20.159 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-05-15T12:34:20.159 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-base-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:34:20.159 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:34:20.159 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:34:20.159 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-05-15T12:34:20.384 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-common-debuginfo 2024-05-15T12:34:20.989 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:02:56 ago on Wed 15 May 2024 12:31:25 PM UTC. 2024-05-15T12:34:21.123 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-05-15T12:34:21.123 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:34:21.123 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-05-15T12:34:21.123 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:34:21.123 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-05-15T12:34:21.123 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 422 M 2024-05-15T12:34:21.123 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:34:21.123 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-05-15T12:34:21.123 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:34:21.123 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-05-15T12:34:21.123 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:34:21.124 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 422 M 2024-05-15T12:34:21.124 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 1.7 G 2024-05-15T12:34:21.124 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-05-15T12:34:49.899 INFO:teuthology.orchestra.run.smithi178.stdout:ceph-common-debuginfo-19.0.0-1882.gae248125.el9 15 MB/s | 422 MB 00:28 2024-05-15T12:34:49.900 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:34:49.900 INFO:teuthology.orchestra.run.smithi178.stdout:Total 15 MB/s | 422 MB 00:28 2024-05-15T12:34:49.901 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-05-15T12:34:49.905 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-05-15T12:34:49.905 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-05-15T12:34:52.411 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-05-15T12:34:52.411 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-05-15T12:35:00.523 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-05-15T12:35:00.629 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1882.gae248125.el9.x8 1/1 2024-05-15T12:35:08.513 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1882.gae248125.el9.x8 1/1 2024-05-15T12:35:08.753 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1882.gae248125.el9.x8 1/1 2024-05-15T12:35:08.753 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:35:08.753 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-05-15T12:35:08.753 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-common-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:35:08.753 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:35:08.753 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-05-15T12:35:08.976 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-05-15T12:35:09.557 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:03:46 ago on Wed 15 May 2024 12:31:23 PM UTC. 2024-05-15T12:35:09.680 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-05-15T12:35:09.681 INFO:teuthology.orchestra.run.smithi117.stdout:======================================================================================== 2024-05-15T12:35:09.681 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-05-15T12:35:09.681 INFO:teuthology.orchestra.run.smithi117.stdout:======================================================================================== 2024-05-15T12:35:09.681 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-05-15T12:35:09.681 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 1.4 M 2024-05-15T12:35:09.681 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:35:09.681 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-05-15T12:35:09.681 INFO:teuthology.orchestra.run.smithi117.stdout:======================================================================================== 2024-05-15T12:35:09.681 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-05-15T12:35:09.681 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:35:09.681 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 1.4 M 2024-05-15T12:35:09.681 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 6.5 M 2024-05-15T12:35:09.681 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-05-15T12:35:10.153 INFO:teuthology.orchestra.run.smithi117.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-18 3.0 MB/s | 1.4 MB 00:00 2024-05-15T12:35:10.153 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:35:10.153 INFO:teuthology.orchestra.run.smithi117.stdout:Total 3.0 MB/s | 1.4 MB 00:00 2024-05-15T12:35:10.154 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-05-15T12:35:10.157 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-05-15T12:35:10.157 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-05-15T12:35:10.199 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-05-15T12:35:10.199 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-05-15T12:35:10.461 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-05-15T12:35:10.568 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1882. 1/1 2024-05-15T12:35:10.982 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1882. 1/1 2024-05-15T12:35:11.211 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1882. 1/1 2024-05-15T12:35:11.212 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:35:11.212 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-05-15T12:35:11.212 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:35:11.212 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:35:11.212 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-05-15T12:35:11.367 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-radosgw-debuginfo 2024-05-15T12:35:11.951 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:03:48 ago on Wed 15 May 2024 12:31:23 PM UTC. 2024-05-15T12:35:12.072 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-05-15T12:35:12.073 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:35:12.073 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-05-15T12:35:12.073 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:35:12.073 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-05-15T12:35:12.073 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 176 M 2024-05-15T12:35:12.073 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:35:12.074 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-05-15T12:35:12.074 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:35:12.074 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-05-15T12:35:12.074 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:35:12.074 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 176 M 2024-05-15T12:35:12.074 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 757 M 2024-05-15T12:35:12.074 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-05-15T12:35:21.554 INFO:teuthology.orchestra.run.smithi117.stdout:ceph-radosgw-debuginfo-19.0.0-1882.gae248125.el 19 MB/s | 176 MB 00:09 2024-05-15T12:35:21.555 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:35:21.555 INFO:teuthology.orchestra.run.smithi117.stdout:Total 19 MB/s | 176 MB 00:09 2024-05-15T12:35:21.556 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-05-15T12:35:21.560 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-05-15T12:35:21.560 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-05-15T12:35:22.625 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-05-15T12:35:22.625 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-05-15T12:35:35.403 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-05-15T12:35:35.495 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1882.gae248125.el9.x8 1/1 2024-05-15T12:35:40.959 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-05-15T12:35:41.118 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1882.gae248125.el9.x 1/1 2024-05-15T12:35:43.682 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1882.gae248125.el9.x8 1/1 2024-05-15T12:35:43.893 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1882.gae248125.el9.x8 1/1 2024-05-15T12:35:43.894 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:35:43.894 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-05-15T12:35:43.894 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-common-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:35:43.894 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:35:43.894 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-05-15T12:35:44.133 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-05-15T12:35:44.713 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:04:19 ago on Wed 15 May 2024 12:31:25 PM UTC. 2024-05-15T12:35:44.836 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-05-15T12:35:44.837 INFO:teuthology.orchestra.run.smithi178.stdout:======================================================================================== 2024-05-15T12:35:44.837 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-05-15T12:35:44.837 INFO:teuthology.orchestra.run.smithi178.stdout:======================================================================================== 2024-05-15T12:35:44.837 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-05-15T12:35:44.837 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 1.4 M 2024-05-15T12:35:44.837 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:35:44.837 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-05-15T12:35:44.837 INFO:teuthology.orchestra.run.smithi178.stdout:======================================================================================== 2024-05-15T12:35:44.837 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-05-15T12:35:44.837 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:35:44.837 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 1.4 M 2024-05-15T12:35:44.837 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 6.5 M 2024-05-15T12:35:44.838 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-05-15T12:35:45.159 INFO:teuthology.orchestra.run.smithi178.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-18 4.5 MB/s | 1.4 MB 00:00 2024-05-15T12:35:45.159 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:35:45.160 INFO:teuthology.orchestra.run.smithi178.stdout:Total 4.4 MB/s | 1.4 MB 00:00 2024-05-15T12:35:45.160 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-05-15T12:35:45.163 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-05-15T12:35:45.163 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-05-15T12:35:45.206 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-05-15T12:35:45.206 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-05-15T12:35:45.464 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-05-15T12:35:45.576 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1882. 1/1 2024-05-15T12:35:45.812 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1882.gae248125.el9.x 1/1 2024-05-15T12:35:45.979 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1882. 1/1 2024-05-15T12:35:46.030 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1882.gae248125.el9.x 1/1 2024-05-15T12:35:46.030 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:35:46.030 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-05-15T12:35:46.030 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-radosgw-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:35:46.030 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:35:46.030 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-05-15T12:35:46.193 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1882. 1/1 2024-05-15T12:35:46.193 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:35:46.193 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-05-15T12:35:46.194 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:35:46.194 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:35:46.194 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-05-15T12:35:46.220 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-test-debuginfo 2024-05-15T12:35:46.359 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-radosgw-debuginfo 2024-05-15T12:35:46.829 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:04:23 ago on Wed 15 May 2024 12:31:23 PM UTC. 2024-05-15T12:35:46.939 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:04:21 ago on Wed 15 May 2024 12:31:25 PM UTC. 2024-05-15T12:35:46.956 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-05-15T12:35:46.957 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:35:46.957 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-05-15T12:35:46.957 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:35:46.957 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-05-15T12:35:46.957 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 675 M 2024-05-15T12:35:46.957 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:35:46.957 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-05-15T12:35:46.957 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:35:46.958 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-05-15T12:35:46.958 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:35:46.958 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 675 M 2024-05-15T12:35:46.958 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 3.1 G 2024-05-15T12:35:46.958 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-05-15T12:35:47.062 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-05-15T12:35:47.063 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:35:47.063 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-05-15T12:35:47.063 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:35:47.063 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-05-15T12:35:47.063 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 176 M 2024-05-15T12:35:47.063 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:35:47.063 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-05-15T12:35:47.063 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:35:47.063 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-05-15T12:35:47.063 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:35:47.064 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 176 M 2024-05-15T12:35:47.064 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 757 M 2024-05-15T12:35:47.064 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-05-15T12:36:21.579 INFO:teuthology.orchestra.run.smithi117.stdout:ceph-test-debuginfo-19.0.0-1882.gae248125.el9.x 20 MB/s | 675 MB 00:34 2024-05-15T12:36:21.580 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:36:21.580 INFO:teuthology.orchestra.run.smithi117.stdout:Total 19 MB/s | 675 MB 00:34 2024-05-15T12:36:21.580 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-05-15T12:36:21.586 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-05-15T12:36:21.586 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-05-15T12:36:22.717 INFO:teuthology.orchestra.run.smithi178.stdout:ceph-radosgw-debuginfo-19.0.0-1882.gae248125.el 4.9 MB/s | 176 MB 00:35 2024-05-15T12:36:22.718 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:36:22.718 INFO:teuthology.orchestra.run.smithi178.stdout:Total 4.9 MB/s | 176 MB 00:35 2024-05-15T12:36:22.719 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-05-15T12:36:22.723 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-05-15T12:36:22.723 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-05-15T12:36:23.787 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-05-15T12:36:23.788 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-05-15T12:36:25.566 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-05-15T12:36:25.566 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-05-15T12:36:42.184 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-05-15T12:36:42.390 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1882.gae248125.el9.x 1/1 2024-05-15T12:36:47.144 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1882.gae248125.el9.x 1/1 2024-05-15T12:36:47.357 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1882.gae248125.el9.x 1/1 2024-05-15T12:36:47.358 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:36:47.358 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-05-15T12:36:47.358 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-radosgw-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:36:47.358 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:36:47.358 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-05-15T12:36:47.573 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-test-debuginfo 2024-05-15T12:36:48.153 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:05:23 ago on Wed 15 May 2024 12:31:25 PM UTC. 2024-05-15T12:36:48.277 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-05-15T12:36:48.278 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:36:48.278 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-05-15T12:36:48.278 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:36:48.278 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-05-15T12:36:48.278 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 675 M 2024-05-15T12:36:48.278 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:36:48.278 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-05-15T12:36:48.278 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:36:48.279 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-05-15T12:36:48.279 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:36:48.279 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 675 M 2024-05-15T12:36:48.279 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 3.1 G 2024-05-15T12:36:48.279 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-05-15T12:37:19.074 INFO:teuthology.orchestra.run.smithi178.stdout:ceph-test-debuginfo-19.0.0-1882.gae248125.el9.x 22 MB/s | 675 MB 00:30 2024-05-15T12:37:19.075 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:37:19.075 INFO:teuthology.orchestra.run.smithi178.stdout:Total 22 MB/s | 675 MB 00:30 2024-05-15T12:37:19.075 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-05-15T12:37:19.081 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-05-15T12:37:19.081 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-05-15T12:37:23.073 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-05-15T12:37:23.074 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-05-15T12:37:37.041 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-05-15T12:37:37.241 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1882.gae248125.el9.x86_ 1/1 2024-05-15T12:37:46.807 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1882.gae248125.el9.x86_ 1/1 2024-05-15T12:37:47.046 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1882.gae248125.el9.x86_ 1/1 2024-05-15T12:37:47.047 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:37:47.047 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-05-15T12:37:47.047 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-test-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:37:47.047 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:37:47.047 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-05-15T12:37:47.321 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-base-debuginfo 2024-05-15T12:37:47.898 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:06:24 ago on Wed 15 May 2024 12:31:23 PM UTC. 2024-05-15T12:37:47.979 INFO:teuthology.orchestra.run.smithi117.stdout:Package ceph-base-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 is already installed. 2024-05-15T12:37:48.020 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-05-15T12:37:48.021 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-05-15T12:37:48.021 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-05-15T12:37:48.078 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-mgr-debuginfo 2024-05-15T12:37:48.654 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:06:25 ago on Wed 15 May 2024 12:31:23 PM UTC. 2024-05-15T12:37:48.778 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-05-15T12:37:48.778 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:37:48.778 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-05-15T12:37:48.778 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:37:48.778 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-05-15T12:37:48.778 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 20 M 2024-05-15T12:37:48.778 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:37:48.778 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-05-15T12:37:48.779 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:37:48.779 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-05-15T12:37:48.779 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:37:48.779 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 20 M 2024-05-15T12:37:48.779 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 104 M 2024-05-15T12:37:48.779 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-05-15T12:37:50.145 INFO:teuthology.orchestra.run.smithi117.stdout:ceph-mgr-debuginfo-19.0.0-1882.gae248125.el9.x8 15 MB/s | 20 MB 00:01 2024-05-15T12:37:50.146 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:37:50.146 INFO:teuthology.orchestra.run.smithi117.stdout:Total 15 MB/s | 20 MB 00:01 2024-05-15T12:37:50.146 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-05-15T12:37:50.150 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-05-15T12:37:50.150 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-05-15T12:37:50.303 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-05-15T12:37:50.303 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-05-15T12:37:52.582 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-05-15T12:37:52.685 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:37:53.611 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:37:53.834 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:37:53.834 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:37:53.834 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-05-15T12:37:53.834 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:37:53.835 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:37:53.835 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-05-15T12:37:54.009 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-mds-debuginfo 2024-05-15T12:37:54.584 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:06:31 ago on Wed 15 May 2024 12:31:23 PM UTC. 2024-05-15T12:37:54.705 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-05-15T12:37:54.706 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:37:54.706 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-05-15T12:37:54.706 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:37:54.706 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-05-15T12:37:54.706 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 49 M 2024-05-15T12:37:54.706 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:37:54.706 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-05-15T12:37:54.706 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:37:54.707 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-05-15T12:37:54.707 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:37:54.707 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 49 M 2024-05-15T12:37:54.707 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 177 M 2024-05-15T12:37:54.707 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-05-15T12:37:57.031 INFO:teuthology.orchestra.run.smithi117.stdout:ceph-mds-debuginfo-19.0.0-1882.gae248125.el9.x8 21 MB/s | 49 MB 00:02 2024-05-15T12:37:57.032 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:37:57.032 INFO:teuthology.orchestra.run.smithi117.stdout:Total 21 MB/s | 49 MB 00:02 2024-05-15T12:37:57.032 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-05-15T12:37:57.036 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-05-15T12:37:57.036 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-05-15T12:37:57.354 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-05-15T12:37:57.354 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-05-15T12:38:02.176 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-05-15T12:38:02.268 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:38:03.585 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:38:03.807 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:38:03.808 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:38:03.808 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-05-15T12:38:03.808 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mds-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:38:03.808 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:38:03.808 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-05-15T12:38:03.988 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-mon-debuginfo 2024-05-15T12:38:04.561 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:06:41 ago on Wed 15 May 2024 12:31:23 PM UTC. 2024-05-15T12:38:04.682 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-05-15T12:38:04.683 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:38:04.683 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-05-15T12:38:04.683 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:38:04.683 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-05-15T12:38:04.683 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 99 M 2024-05-15T12:38:04.683 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:38:04.683 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-05-15T12:38:04.683 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:38:04.683 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-05-15T12:38:04.684 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:38:04.684 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 99 M 2024-05-15T12:38:04.684 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 450 M 2024-05-15T12:38:04.684 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-05-15T12:38:09.332 INFO:teuthology.orchestra.run.smithi117.stdout:ceph-mon-debuginfo-19.0.0-1882.gae248125.el9.x8 21 MB/s | 99 MB 00:04 2024-05-15T12:38:09.333 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:38:09.333 INFO:teuthology.orchestra.run.smithi117.stdout:Total 21 MB/s | 99 MB 00:04 2024-05-15T12:38:09.334 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-05-15T12:38:09.338 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-05-15T12:38:09.338 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-05-15T12:38:09.954 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-05-15T12:38:09.954 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-05-15T12:38:20.254 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-05-15T12:38:20.352 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:38:23.238 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:38:23.485 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:38:23.485 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:38:23.485 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-05-15T12:38:23.486 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mon-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:38:23.486 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:38:23.486 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-05-15T12:38:23.679 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-osd-debuginfo 2024-05-15T12:38:24.254 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:07:01 ago on Wed 15 May 2024 12:31:23 PM UTC. 2024-05-15T12:38:24.376 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-05-15T12:38:24.377 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:38:24.377 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-05-15T12:38:24.377 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:38:24.377 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-05-15T12:38:24.377 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 289 M 2024-05-15T12:38:24.377 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:38:24.377 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-05-15T12:38:24.377 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:38:24.377 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-05-15T12:38:24.378 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:38:24.378 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 289 M 2024-05-15T12:38:24.378 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 1.2 G 2024-05-15T12:38:24.378 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-05-15T12:38:34.515 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-05-15T12:38:34.697 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1882.gae248125.el9.x86_ 1/1 2024-05-15T12:38:36.841 INFO:teuthology.orchestra.run.smithi117.stdout:ceph-osd-debuginfo-19.0.0-1882.gae248125.el9.x8 23 MB/s | 289 MB 00:12 2024-05-15T12:38:36.842 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:38:36.843 INFO:teuthology.orchestra.run.smithi117.stdout:Total 23 MB/s | 289 MB 00:12 2024-05-15T12:38:36.843 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-05-15T12:38:36.847 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-05-15T12:38:36.847 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-05-15T12:38:38.569 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-05-15T12:38:38.569 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-05-15T12:38:44.543 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1882.gae248125.el9.x86_ 1/1 2024-05-15T12:38:44.819 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1882.gae248125.el9.x86_ 1/1 2024-05-15T12:38:44.819 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:38:44.819 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-05-15T12:38:44.819 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-test-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:38:44.819 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:38:44.819 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-05-15T12:38:45.095 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-base-debuginfo 2024-05-15T12:38:45.667 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:07:20 ago on Wed 15 May 2024 12:31:25 PM UTC. 2024-05-15T12:38:45.748 INFO:teuthology.orchestra.run.smithi178.stdout:Package ceph-base-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 is already installed. 2024-05-15T12:38:45.789 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-05-15T12:38:45.790 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-05-15T12:38:45.791 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-05-15T12:38:45.859 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-mgr-debuginfo 2024-05-15T12:38:46.432 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:07:21 ago on Wed 15 May 2024 12:31:25 PM UTC. 2024-05-15T12:38:46.556 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-05-15T12:38:46.556 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:38:46.556 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-05-15T12:38:46.556 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:38:46.556 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-05-15T12:38:46.556 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 20 M 2024-05-15T12:38:46.556 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:38:46.556 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-05-15T12:38:46.557 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:38:46.557 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-05-15T12:38:46.557 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:38:46.557 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 20 M 2024-05-15T12:38:46.557 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 104 M 2024-05-15T12:38:46.557 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-05-15T12:38:50.284 INFO:teuthology.orchestra.run.smithi178.stdout:ceph-mgr-debuginfo-19.0.0-1882.gae248125.el9.x8 5.4 MB/s | 20 MB 00:03 2024-05-15T12:38:50.285 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:38:50.286 INFO:teuthology.orchestra.run.smithi178.stdout:Total 5.4 MB/s | 20 MB 00:03 2024-05-15T12:38:50.286 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-05-15T12:38:50.290 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-05-15T12:38:50.290 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-05-15T12:38:50.441 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-05-15T12:38:50.442 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-05-15T12:38:52.716 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-05-15T12:38:52.824 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:38:53.810 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:38:54.044 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:38:54.044 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:38:54.044 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-05-15T12:38:54.044 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:38:54.044 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:38:54.044 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-05-15T12:38:54.218 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-mds-debuginfo 2024-05-15T12:38:54.791 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:07:29 ago on Wed 15 May 2024 12:31:25 PM UTC. 2024-05-15T12:38:54.914 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-05-15T12:38:54.915 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:38:54.915 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-05-15T12:38:54.915 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:38:54.915 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-05-15T12:38:54.915 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 49 M 2024-05-15T12:38:54.915 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:38:54.915 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-05-15T12:38:54.915 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:38:54.916 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-05-15T12:38:54.916 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:38:54.916 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 49 M 2024-05-15T12:38:54.916 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 177 M 2024-05-15T12:38:54.916 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-05-15T12:38:57.403 INFO:teuthology.orchestra.run.smithi178.stdout:ceph-mds-debuginfo-19.0.0-1882.gae248125.el9.x8 20 MB/s | 49 MB 00:02 2024-05-15T12:38:57.404 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:38:57.405 INFO:teuthology.orchestra.run.smithi178.stdout:Total 20 MB/s | 49 MB 00:02 2024-05-15T12:38:57.406 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-05-15T12:38:57.410 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-05-15T12:38:57.410 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-05-15T12:38:57.730 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-05-15T12:38:57.730 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-05-15T12:39:02.572 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-05-15T12:39:02.658 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:39:04.013 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:39:04.241 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:39:04.241 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:39:04.241 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-05-15T12:39:04.241 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mds-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:39:04.241 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:39:04.241 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-05-15T12:39:04.418 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-mon-debuginfo 2024-05-15T12:39:04.992 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:07:40 ago on Wed 15 May 2024 12:31:25 PM UTC. 2024-05-15T12:39:05.114 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-05-15T12:39:05.114 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:39:05.115 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-05-15T12:39:05.115 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:39:05.115 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-05-15T12:39:05.115 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 99 M 2024-05-15T12:39:05.115 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:39:05.115 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-05-15T12:39:05.115 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:39:05.115 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-05-15T12:39:05.115 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:39:05.115 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 99 M 2024-05-15T12:39:05.115 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 450 M 2024-05-15T12:39:05.115 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-05-15T12:39:08.171 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-05-15T12:39:08.269 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:39:09.982 INFO:teuthology.orchestra.run.smithi178.stdout:ceph-mon-debuginfo-19.0.0-1882.gae248125.el9.x8 20 MB/s | 99 MB 00:04 2024-05-15T12:39:09.983 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:39:09.983 INFO:teuthology.orchestra.run.smithi178.stdout:Total 20 MB/s | 99 MB 00:04 2024-05-15T12:39:09.984 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-05-15T12:39:09.988 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-05-15T12:39:09.988 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-05-15T12:39:10.601 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-05-15T12:39:10.602 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-05-15T12:39:15.653 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:39:15.884 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:39:15.884 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:39:15.884 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-05-15T12:39:15.884 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-osd-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:39:15.884 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:39:15.884 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-05-15T12:39:16.101 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-fuse-debuginfo 2024-05-15T12:39:16.679 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:07:53 ago on Wed 15 May 2024 12:31:23 PM UTC. 2024-05-15T12:39:16.802 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-05-15T12:39:16.803 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:39:16.803 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-05-15T12:39:16.803 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:39:16.803 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-05-15T12:39:16.803 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 6.0 M 2024-05-15T12:39:16.803 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:39:16.803 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-05-15T12:39:16.803 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:39:16.803 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-05-15T12:39:16.803 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:39:16.804 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 6.0 M 2024-05-15T12:39:16.804 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 28 M 2024-05-15T12:39:16.804 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-05-15T12:39:17.514 INFO:teuthology.orchestra.run.smithi117.stdout:ceph-fuse-debuginfo-19.0.0-1882.gae248125.el9.x 8.4 MB/s | 6.0 MB 00:00 2024-05-15T12:39:17.514 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:39:17.515 INFO:teuthology.orchestra.run.smithi117.stdout:Total 8.4 MB/s | 6.0 MB 00:00 2024-05-15T12:39:17.515 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-05-15T12:39:17.519 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-05-15T12:39:17.519 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-05-15T12:39:17.589 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-05-15T12:39:17.589 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-05-15T12:39:18.321 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-05-15T12:39:18.411 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1882.gae248125.el9.x86_ 1/1 2024-05-15T12:39:18.909 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1882.gae248125.el9.x86_ 1/1 2024-05-15T12:39:19.148 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1882.gae248125.el9.x86_ 1/1 2024-05-15T12:39:19.149 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:39:19.149 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-05-15T12:39:19.149 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-fuse-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:39:19.149 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:39:19.149 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-05-15T12:39:19.332 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install librados-devel-debuginfo 2024-05-15T12:39:19.912 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:07:56 ago on Wed 15 May 2024 12:31:23 PM UTC. 2024-05-15T12:39:20.033 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-05-15T12:39:20.034 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:39:20.034 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-05-15T12:39:20.034 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:39:20.034 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-05-15T12:39:20.034 INFO:teuthology.orchestra.run.smithi117.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 109 k 2024-05-15T12:39:20.035 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:39:20.035 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-05-15T12:39:20.035 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:39:20.035 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-05-15T12:39:20.035 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:39:20.035 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 109 k 2024-05-15T12:39:20.035 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 440 k 2024-05-15T12:39:20.035 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-05-15T12:39:20.260 INFO:teuthology.orchestra.run.smithi117.stdout:librados-devel-debuginfo-19.0.0-1882.gae248125. 482 kB/s | 109 kB 00:00 2024-05-15T12:39:20.261 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:39:20.261 INFO:teuthology.orchestra.run.smithi117.stdout:Total 479 kB/s | 109 kB 00:00 2024-05-15T12:39:20.262 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-05-15T12:39:20.265 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-05-15T12:39:20.265 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-05-15T12:39:20.301 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-05-15T12:39:20.301 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-05-15T12:39:20.419 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-05-15T12:39:20.517 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1882.gae248125.el9 1/1 2024-05-15T12:39:20.871 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1882.gae248125.el9 1/1 2024-05-15T12:39:20.942 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-05-15T12:39:21.045 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:39:21.098 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1882.gae248125.el9 1/1 2024-05-15T12:39:21.098 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:39:21.098 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-05-15T12:39:21.098 INFO:teuthology.orchestra.run.smithi117.stdout: librados-devel-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:39:21.098 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:39:21.099 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-05-15T12:39:21.300 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install libcephfs2-debuginfo 2024-05-15T12:39:21.876 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:07:58 ago on Wed 15 May 2024 12:31:23 PM UTC. 2024-05-15T12:39:21.997 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-05-15T12:39:21.997 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:39:21.997 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-05-15T12:39:21.998 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:39:21.998 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-05-15T12:39:21.998 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 5.4 M 2024-05-15T12:39:21.998 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:39:21.998 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-05-15T12:39:21.998 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:39:21.998 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-05-15T12:39:21.998 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:39:21.998 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 5.4 M 2024-05-15T12:39:21.998 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 25 M 2024-05-15T12:39:21.998 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-05-15T12:39:22.905 INFO:teuthology.orchestra.run.smithi117.stdout:libcephfs2-debuginfo-19.0.0-1882.gae248125.el9. 5.9 MB/s | 5.4 MB 00:00 2024-05-15T12:39:22.906 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:39:22.906 INFO:teuthology.orchestra.run.smithi117.stdout:Total 5.9 MB/s | 5.4 MB 00:00 2024-05-15T12:39:22.907 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-05-15T12:39:22.910 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-05-15T12:39:22.910 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-05-15T12:39:22.976 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-05-15T12:39:22.977 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-05-15T12:39:23.646 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-05-15T12:39:23.745 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1882.gae248125.el9.x86 1/1 2024-05-15T12:39:24.000 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:39:24.244 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:39:24.244 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:39:24.244 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-05-15T12:39:24.244 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mon-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:39:24.244 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:39:24.244 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-05-15T12:39:24.249 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1882.gae248125.el9.x86 1/1 2024-05-15T12:39:24.424 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-osd-debuginfo 2024-05-15T12:39:24.470 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1882.gae248125.el9.x86 1/1 2024-05-15T12:39:24.471 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:39:24.471 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-05-15T12:39:24.471 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs2-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:39:24.471 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:39:24.471 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-05-15T12:39:24.651 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install librados2-debuginfo 2024-05-15T12:39:25.001 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:08:00 ago on Wed 15 May 2024 12:31:25 PM UTC. 2024-05-15T12:39:25.123 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-05-15T12:39:25.124 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:39:25.124 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-05-15T12:39:25.124 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:39:25.124 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-05-15T12:39:25.124 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 289 M 2024-05-15T12:39:25.124 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:39:25.124 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-05-15T12:39:25.124 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:39:25.125 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-05-15T12:39:25.125 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:39:25.125 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 289 M 2024-05-15T12:39:25.125 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 1.2 G 2024-05-15T12:39:25.125 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-05-15T12:39:25.229 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:08:02 ago on Wed 15 May 2024 12:31:23 PM UTC. 2024-05-15T12:39:25.354 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-05-15T12:39:25.354 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:39:25.355 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-05-15T12:39:25.355 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:39:25.355 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-05-15T12:39:25.355 INFO:teuthology.orchestra.run.smithi117.stdout: librados2-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 45 M 2024-05-15T12:39:25.355 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:39:25.355 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-05-15T12:39:25.355 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:39:25.355 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-05-15T12:39:25.355 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:39:25.355 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 45 M 2024-05-15T12:39:25.355 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 255 M 2024-05-15T12:39:25.355 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-05-15T12:39:37.572 INFO:teuthology.orchestra.run.smithi117.stdout:librados2-debuginfo-19.0.0-1882.gae248125.el9.x 3.7 MB/s | 45 MB 00:12 2024-05-15T12:39:37.573 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:39:37.574 INFO:teuthology.orchestra.run.smithi117.stdout:Total 3.7 MB/s | 45 MB 00:12 2024-05-15T12:39:37.574 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-05-15T12:39:37.578 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-05-15T12:39:37.578 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-05-15T12:39:37.882 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-05-15T12:39:37.883 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-05-15T12:39:39.182 INFO:teuthology.orchestra.run.smithi178.stdout:ceph-osd-debuginfo-19.0.0-1882.gae248125.el9.x8 21 MB/s | 289 MB 00:14 2024-05-15T12:39:39.183 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:39:39.184 INFO:teuthology.orchestra.run.smithi178.stdout:Total 21 MB/s | 289 MB 00:14 2024-05-15T12:39:39.184 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-05-15T12:39:39.188 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-05-15T12:39:39.188 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-05-15T12:39:40.913 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-05-15T12:39:40.913 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-05-15T12:39:43.007 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-05-15T12:39:43.086 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librados2-debuginfo-2:19.0.0-1882.gae248125.el9.x86_ 1/1 2024-05-15T12:39:44.895 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1882.gae248125.el9.x86_ 1/1 2024-05-15T12:39:45.134 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librados2-debuginfo-2:19.0.0-1882.gae248125.el9.x86_ 1/1 2024-05-15T12:39:45.134 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:39:45.134 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-05-15T12:39:45.134 INFO:teuthology.orchestra.run.smithi117.stdout: librados2-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:39:45.135 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:39:45.135 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-05-15T12:39:45.342 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install librbd1-debuginfo 2024-05-15T12:39:45.925 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:08:22 ago on Wed 15 May 2024 12:31:23 PM UTC. 2024-05-15T12:39:46.048 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-05-15T12:39:46.049 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:39:46.049 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-05-15T12:39:46.049 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:39:46.049 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-05-15T12:39:46.049 INFO:teuthology.orchestra.run.smithi117.stdout: librbd1-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 52 M 2024-05-15T12:39:46.049 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:39:46.049 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-05-15T12:39:46.049 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:39:46.049 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-05-15T12:39:46.049 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:39:46.050 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 52 M 2024-05-15T12:39:46.050 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 308 M 2024-05-15T12:39:46.050 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-05-15T12:39:48.533 INFO:teuthology.orchestra.run.smithi117.stdout:librbd1-debuginfo-19.0.0-1882.gae248125.el9.x86 21 MB/s | 52 MB 00:02 2024-05-15T12:39:48.534 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:39:48.534 INFO:teuthology.orchestra.run.smithi117.stdout:Total 21 MB/s | 52 MB 00:02 2024-05-15T12:39:48.534 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-05-15T12:39:48.538 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-05-15T12:39:48.538 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-05-15T12:39:48.883 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-05-15T12:39:48.883 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-05-15T12:39:54.920 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-05-15T12:39:55.021 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librbd1-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:39:57.116 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:39:57.358 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:39:57.358 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:39:57.358 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-05-15T12:39:57.358 INFO:teuthology.orchestra.run.smithi117.stdout: librbd1-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:39:57.358 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:39:57.359 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-05-15T12:39:57.671 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install python3-cephfs-debuginfo 2024-05-15T12:39:58.246 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:08:35 ago on Wed 15 May 2024 12:31:23 PM UTC. 2024-05-15T12:39:58.370 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-05-15T12:39:58.370 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:39:58.370 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-05-15T12:39:58.371 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:39:58.371 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-05-15T12:39:58.371 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 615 k 2024-05-15T12:39:58.371 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:39:58.371 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-05-15T12:39:58.371 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:39:58.371 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-05-15T12:39:58.371 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:39:58.372 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 615 k 2024-05-15T12:39:58.372 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 2.1 M 2024-05-15T12:39:58.372 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-05-15T12:39:58.909 INFO:teuthology.orchestra.run.smithi117.stdout:python3-cephfs-debuginfo-19.0.0-1882.gae248125. 1.1 MB/s | 615 kB 00:00 2024-05-15T12:39:58.910 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:39:58.910 INFO:teuthology.orchestra.run.smithi117.stdout:Total 1.1 MB/s | 615 kB 00:00 2024-05-15T12:39:58.910 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-05-15T12:39:58.914 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-05-15T12:39:58.914 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-05-15T12:39:58.953 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-05-15T12:39:58.953 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-05-15T12:39:59.119 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-05-15T12:39:59.215 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1882.gae248125.el9 1/1 2024-05-15T12:39:59.567 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1882.gae248125.el9 1/1 2024-05-15T12:39:59.797 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1882.gae248125.el9 1/1 2024-05-15T12:39:59.797 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:39:59.797 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-05-15T12:39:59.798 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cephfs-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:39:59.798 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:39:59.798 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-05-15T12:39:59.981 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install python3-rados-debuginfo 2024-05-15T12:40:00.558 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:08:37 ago on Wed 15 May 2024 12:31:23 PM UTC. 2024-05-15T12:40:00.682 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-05-15T12:40:00.683 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:40:00.683 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-05-15T12:40:00.683 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:40:00.683 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-05-15T12:40:00.683 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 769 k 2024-05-15T12:40:00.683 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:40:00.683 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-05-15T12:40:00.683 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:40:00.683 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-05-15T12:40:00.683 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:40:00.683 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 769 k 2024-05-15T12:40:00.683 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 2.7 M 2024-05-15T12:40:00.683 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-05-15T12:40:01.537 INFO:teuthology.orchestra.run.smithi117.stdout:python3-rados-debuginfo-19.0.0-1882.gae248125.e 901 kB/s | 769 kB 00:00 2024-05-15T12:40:01.538 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:40:01.539 INFO:teuthology.orchestra.run.smithi117.stdout:Total 899 kB/s | 769 kB 00:00 2024-05-15T12:40:01.539 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-05-15T12:40:01.543 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-05-15T12:40:01.543 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-05-15T12:40:01.585 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-05-15T12:40:01.585 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-05-15T12:40:01.769 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-05-15T12:40:01.852 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1882.gae248125.el9. 1/1 2024-05-15T12:40:02.185 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1882.gae248125.el9. 1/1 2024-05-15T12:40:02.414 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1882.gae248125.el9. 1/1 2024-05-15T12:40:02.414 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:40:02.414 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-05-15T12:40:02.414 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rados-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:40:02.414 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:40:02.414 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-05-15T12:40:02.584 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install python3-rbd-debuginfo 2024-05-15T12:40:03.159 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:08:40 ago on Wed 15 May 2024 12:31:23 PM UTC. 2024-05-15T12:40:03.281 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-05-15T12:40:03.282 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:40:03.282 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-05-15T12:40:03.282 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:40:03.282 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-05-15T12:40:03.282 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 1.1 M 2024-05-15T12:40:03.282 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:40:03.282 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-05-15T12:40:03.282 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:40:03.282 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-05-15T12:40:03.282 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:40:03.282 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 1.1 M 2024-05-15T12:40:03.282 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 4.3 M 2024-05-15T12:40:03.282 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-05-15T12:40:04.407 INFO:teuthology.orchestra.run.smithi117.stdout:python3-rbd-debuginfo-19.0.0-1882.gae248125.el9 1.0 MB/s | 1.1 MB 00:01 2024-05-15T12:40:04.408 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:40:04.408 INFO:teuthology.orchestra.run.smithi117.stdout:Total 1.0 MB/s | 1.1 MB 00:01 2024-05-15T12:40:04.409 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-05-15T12:40:04.412 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-05-15T12:40:04.413 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-05-15T12:40:04.457 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-05-15T12:40:04.457 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-05-15T12:40:04.685 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-05-15T12:40:04.788 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1882.gae248125.el9.x8 1/1 2024-05-15T12:40:05.175 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1882.gae248125.el9.x8 1/1 2024-05-15T12:40:05.396 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1882.gae248125.el9.x8 1/1 2024-05-15T12:40:05.396 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:40:05.396 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-05-15T12:40:05.396 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rbd-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:40:05.396 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:40:05.396 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-05-15T12:40:05.588 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install python3-rgw-debuginfo 2024-05-15T12:40:06.162 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:08:43 ago on Wed 15 May 2024 12:31:23 PM UTC. 2024-05-15T12:40:06.284 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-05-15T12:40:06.285 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:40:06.285 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-05-15T12:40:06.286 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:40:06.286 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-05-15T12:40:06.286 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 337 k 2024-05-15T12:40:06.286 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:40:06.286 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-05-15T12:40:06.286 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:40:06.286 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-05-15T12:40:06.286 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:40:06.286 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 337 k 2024-05-15T12:40:06.286 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 1.0 M 2024-05-15T12:40:06.286 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-05-15T12:40:06.752 INFO:teuthology.orchestra.run.smithi117.stdout:python3-rgw-debuginfo-19.0.0-1882.gae248125.el9 725 kB/s | 337 kB 00:00 2024-05-15T12:40:06.752 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:40:06.752 INFO:teuthology.orchestra.run.smithi117.stdout:Total 723 kB/s | 337 kB 00:00 2024-05-15T12:40:06.753 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-05-15T12:40:06.756 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-05-15T12:40:06.756 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-05-15T12:40:06.793 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-05-15T12:40:06.793 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-05-15T12:40:06.934 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-05-15T12:40:07.041 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1882.gae248125.el9.x8 1/1 2024-05-15T12:40:07.387 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1882.gae248125.el9.x8 1/1 2024-05-15T12:40:07.620 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1882.gae248125.el9.x8 1/1 2024-05-15T12:40:07.620 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:40:07.620 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-05-15T12:40:07.621 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rgw-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:40:07.621 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:40:07.621 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-05-15T12:40:07.783 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install rbd-fuse-debuginfo 2024-05-15T12:40:08.363 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:08:45 ago on Wed 15 May 2024 12:31:23 PM UTC. 2024-05-15T12:40:08.486 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-05-15T12:40:08.487 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:40:08.487 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-05-15T12:40:08.487 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:40:08.487 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-05-15T12:40:08.487 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 517 k 2024-05-15T12:40:08.487 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:40:08.487 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-05-15T12:40:08.487 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:40:08.488 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-05-15T12:40:08.488 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:40:08.488 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 517 k 2024-05-15T12:40:08.488 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 2.6 M 2024-05-15T12:40:08.488 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-05-15T12:40:09.088 INFO:teuthology.orchestra.run.smithi117.stdout:rbd-fuse-debuginfo-19.0.0-1882.gae248125.el9.x8 861 kB/s | 517 kB 00:00 2024-05-15T12:40:09.088 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:40:09.089 INFO:teuthology.orchestra.run.smithi117.stdout:Total 859 kB/s | 517 kB 00:00 2024-05-15T12:40:09.089 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-05-15T12:40:09.092 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-05-15T12:40:09.092 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-05-15T12:40:09.131 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-05-15T12:40:09.132 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-05-15T12:40:09.290 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-05-15T12:40:09.394 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:40:09.730 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:40:09.962 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:40:09.963 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:40:09.963 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-05-15T12:40:09.963 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-fuse-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:40:09.963 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:40:09.963 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-05-15T12:40:10.126 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install rbd-mirror-debuginfo 2024-05-15T12:40:10.495 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-05-15T12:40:10.577 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:40:10.705 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:08:47 ago on Wed 15 May 2024 12:31:23 PM UTC. 2024-05-15T12:40:10.828 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-05-15T12:40:10.829 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:40:10.829 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-05-15T12:40:10.829 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:40:10.829 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-05-15T12:40:10.829 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 52 M 2024-05-15T12:40:10.829 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:40:10.829 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-05-15T12:40:10.829 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:40:10.829 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-05-15T12:40:10.829 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:40:10.829 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 52 M 2024-05-15T12:40:10.829 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 298 M 2024-05-15T12:40:10.829 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-05-15T12:40:17.748 INFO:teuthology.orchestra.run.smithi117.stdout:rbd-mirror-debuginfo-19.0.0-1882.gae248125.el9. 7.5 MB/s | 52 MB 00:06 2024-05-15T12:40:17.749 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:40:17.749 INFO:teuthology.orchestra.run.smithi117.stdout:Total 7.5 MB/s | 52 MB 00:06 2024-05-15T12:40:17.749 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-05-15T12:40:17.754 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-05-15T12:40:17.754 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-05-15T12:40:18.093 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-05-15T12:40:18.093 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-05-15T12:40:18.295 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:40:18.519 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:40:18.519 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:40:18.519 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-05-15T12:40:18.519 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-osd-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:40:18.519 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:40:18.519 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-05-15T12:40:18.711 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-fuse-debuginfo 2024-05-15T12:40:19.290 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:08:54 ago on Wed 15 May 2024 12:31:25 PM UTC. 2024-05-15T12:40:19.412 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-05-15T12:40:19.412 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:40:19.412 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-05-15T12:40:19.412 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:40:19.412 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-05-15T12:40:19.413 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 6.0 M 2024-05-15T12:40:19.413 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:40:19.413 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-05-15T12:40:19.413 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:40:19.413 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-05-15T12:40:19.413 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:40:19.413 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 6.0 M 2024-05-15T12:40:19.414 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 28 M 2024-05-15T12:40:19.414 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-05-15T12:40:20.200 INFO:teuthology.orchestra.run.smithi178.stdout:ceph-fuse-debuginfo-19.0.0-1882.gae248125.el9.x 7.6 MB/s | 6.0 MB 00:00 2024-05-15T12:40:20.201 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:40:20.201 INFO:teuthology.orchestra.run.smithi178.stdout:Total 7.6 MB/s | 6.0 MB 00:00 2024-05-15T12:40:20.201 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-05-15T12:40:20.205 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-05-15T12:40:20.205 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-05-15T12:40:20.275 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-05-15T12:40:20.275 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-05-15T12:40:21.014 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-05-15T12:40:21.114 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1882.gae248125.el9.x86_ 1/1 2024-05-15T12:40:21.629 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1882.gae248125.el9.x86_ 1/1 2024-05-15T12:40:21.860 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1882.gae248125.el9.x86_ 1/1 2024-05-15T12:40:21.860 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:40:21.860 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-05-15T12:40:21.860 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-fuse-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:40:21.860 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:40:21.860 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-05-15T12:40:22.024 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install librados-devel-debuginfo 2024-05-15T12:40:22.602 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:08:57 ago on Wed 15 May 2024 12:31:25 PM UTC. 2024-05-15T12:40:22.726 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-05-15T12:40:22.726 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:40:22.727 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-05-15T12:40:22.727 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:40:22.727 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-05-15T12:40:22.727 INFO:teuthology.orchestra.run.smithi178.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 109 k 2024-05-15T12:40:22.727 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:40:22.727 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-05-15T12:40:22.727 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:40:22.727 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-05-15T12:40:22.727 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:40:22.727 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 109 k 2024-05-15T12:40:22.727 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 440 k 2024-05-15T12:40:22.727 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-05-15T12:40:22.936 INFO:teuthology.orchestra.run.smithi178.stdout:librados-devel-debuginfo-19.0.0-1882.gae248125. 522 kB/s | 109 kB 00:00 2024-05-15T12:40:22.936 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:40:22.936 INFO:teuthology.orchestra.run.smithi178.stdout:Total 519 kB/s | 109 kB 00:00 2024-05-15T12:40:22.937 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-05-15T12:40:22.940 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-05-15T12:40:22.940 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-05-15T12:40:22.975 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-05-15T12:40:22.976 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-05-15T12:40:23.094 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-05-15T12:40:23.209 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1882.gae248125.el9 1/1 2024-05-15T12:40:23.566 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1882.gae248125.el9 1/1 2024-05-15T12:40:23.794 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1882.gae248125.el9 1/1 2024-05-15T12:40:23.794 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:40:23.794 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-05-15T12:40:23.794 INFO:teuthology.orchestra.run.smithi178.stdout: librados-devel-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:40:23.794 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:40:23.794 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-05-15T12:40:23.986 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install libcephfs2-debuginfo 2024-05-15T12:40:24.023 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-05-15T12:40:24.120 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1882.gae248125.el9.x86 1/1 2024-05-15T12:40:24.573 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:08:59 ago on Wed 15 May 2024 12:31:25 PM UTC. 2024-05-15T12:40:24.695 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-05-15T12:40:24.696 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:40:24.696 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-05-15T12:40:24.696 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:40:24.696 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-05-15T12:40:24.697 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 5.4 M 2024-05-15T12:40:24.697 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:40:24.697 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-05-15T12:40:24.697 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:40:24.697 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-05-15T12:40:24.697 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:40:24.697 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 5.4 M 2024-05-15T12:40:24.697 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 25 M 2024-05-15T12:40:24.697 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-05-15T12:40:25.256 INFO:teuthology.orchestra.run.smithi178.stdout:libcephfs2-debuginfo-19.0.0-1882.gae248125.el9. 9.6 MB/s | 5.4 MB 00:00 2024-05-15T12:40:25.256 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:40:25.257 INFO:teuthology.orchestra.run.smithi178.stdout:Total 9.6 MB/s | 5.4 MB 00:00 2024-05-15T12:40:25.257 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-05-15T12:40:25.261 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-05-15T12:40:25.261 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-05-15T12:40:25.327 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-05-15T12:40:25.327 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-05-15T12:40:25.994 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-05-15T12:40:26.086 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1882.gae248125.el9.x86 1/1 2024-05-15T12:40:26.151 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1882.gae248125.el9.x86 1/1 2024-05-15T12:40:26.392 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1882.gae248125.el9.x86 1/1 2024-05-15T12:40:26.393 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:40:26.393 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-05-15T12:40:26.393 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-mirror-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:40:26.393 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:40:26.393 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-05-15T12:40:26.569 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1882.gae248125.el9.x86 1/1 2024-05-15T12:40:26.586 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install rbd-nbd-debuginfo 2024-05-15T12:40:26.801 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1882.gae248125.el9.x86 1/1 2024-05-15T12:40:26.801 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:40:26.801 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-05-15T12:40:26.801 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs2-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:40:26.801 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:40:26.801 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-05-15T12:40:26.995 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install librados2-debuginfo 2024-05-15T12:40:27.158 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:09:04 ago on Wed 15 May 2024 12:31:23 PM UTC. 2024-05-15T12:40:27.282 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-05-15T12:40:27.282 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:40:27.282 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-05-15T12:40:27.282 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:40:27.282 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-05-15T12:40:27.283 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 1.3 M 2024-05-15T12:40:27.283 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:40:27.283 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-05-15T12:40:27.283 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-05-15T12:40:27.283 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-05-15T12:40:27.283 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:40:27.283 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 1.3 M 2024-05-15T12:40:27.283 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 5.7 M 2024-05-15T12:40:27.284 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-05-15T12:40:27.566 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:09:02 ago on Wed 15 May 2024 12:31:25 PM UTC. 2024-05-15T12:40:27.607 INFO:teuthology.orchestra.run.smithi117.stdout:rbd-nbd-debuginfo-19.0.0-1882.gae248125.el9.x86 3.9 MB/s | 1.3 MB 00:00 2024-05-15T12:40:27.608 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:40:27.608 INFO:teuthology.orchestra.run.smithi117.stdout:Total 3.9 MB/s | 1.3 MB 00:00 2024-05-15T12:40:27.608 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-05-15T12:40:27.612 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-05-15T12:40:27.612 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-05-15T12:40:27.655 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-05-15T12:40:27.656 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-05-15T12:40:27.688 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-05-15T12:40:27.688 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:40:27.689 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-05-15T12:40:27.689 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:40:27.689 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-05-15T12:40:27.689 INFO:teuthology.orchestra.run.smithi178.stdout: librados2-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 45 M 2024-05-15T12:40:27.689 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:40:27.689 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-05-15T12:40:27.689 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:40:27.689 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-05-15T12:40:27.689 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:40:27.690 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 45 M 2024-05-15T12:40:27.690 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 255 M 2024-05-15T12:40:27.690 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-05-15T12:40:27.897 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-05-15T12:40:27.997 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:40:28.482 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:40:28.709 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:40:28.709 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:40:28.709 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-05-15T12:40:28.709 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-nbd-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:40:28.709 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:40:28.709 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-05-15T12:40:28.886 DEBUG:teuthology.parallel:result is None 2024-05-15T12:40:30.491 INFO:teuthology.orchestra.run.smithi178.stdout:librados2-debuginfo-19.0.0-1882.gae248125.el9.x 16 MB/s | 45 MB 00:02 2024-05-15T12:40:30.492 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:40:30.493 INFO:teuthology.orchestra.run.smithi178.stdout:Total 16 MB/s | 45 MB 00:02 2024-05-15T12:40:30.493 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-05-15T12:40:30.497 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-05-15T12:40:30.497 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-05-15T12:40:30.801 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-05-15T12:40:30.801 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-05-15T12:40:35.918 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-05-15T12:40:36.013 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librados2-debuginfo-2:19.0.0-1882.gae248125.el9.x86_ 1/1 2024-05-15T12:40:37.865 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1882.gae248125.el9.x86_ 1/1 2024-05-15T12:40:38.090 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librados2-debuginfo-2:19.0.0-1882.gae248125.el9.x86_ 1/1 2024-05-15T12:40:38.090 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:40:38.090 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-05-15T12:40:38.090 INFO:teuthology.orchestra.run.smithi178.stdout: librados2-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:40:38.090 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:40:38.090 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-05-15T12:40:38.305 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install librbd1-debuginfo 2024-05-15T12:40:38.889 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:09:13 ago on Wed 15 May 2024 12:31:25 PM UTC. 2024-05-15T12:40:39.014 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-05-15T12:40:39.015 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:40:39.015 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-05-15T12:40:39.015 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:40:39.015 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-05-15T12:40:39.015 INFO:teuthology.orchestra.run.smithi178.stdout: librbd1-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 52 M 2024-05-15T12:40:39.015 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:40:39.015 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-05-15T12:40:39.015 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:40:39.015 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-05-15T12:40:39.015 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:40:39.016 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 52 M 2024-05-15T12:40:39.016 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 308 M 2024-05-15T12:40:39.016 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-05-15T12:40:41.702 INFO:teuthology.orchestra.run.smithi178.stdout:librbd1-debuginfo-19.0.0-1882.gae248125.el9.x86 19 MB/s | 52 MB 00:02 2024-05-15T12:40:41.703 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:40:41.703 INFO:teuthology.orchestra.run.smithi178.stdout:Total 19 MB/s | 52 MB 00:02 2024-05-15T12:40:41.704 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-05-15T12:40:41.708 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-05-15T12:40:41.708 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-05-15T12:40:42.051 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-05-15T12:40:42.052 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-05-15T12:40:48.049 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-05-15T12:40:48.166 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librbd1-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:40:50.368 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:40:50.597 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:40:50.597 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:40:50.597 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-05-15T12:40:50.597 INFO:teuthology.orchestra.run.smithi178.stdout: librbd1-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:40:50.597 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:40:50.597 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-05-15T12:40:50.792 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install python3-cephfs-debuginfo 2024-05-15T12:40:51.385 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:09:26 ago on Wed 15 May 2024 12:31:25 PM UTC. 2024-05-15T12:40:51.517 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-05-15T12:40:51.518 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:40:51.518 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-05-15T12:40:51.518 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:40:51.518 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-05-15T12:40:51.518 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 615 k 2024-05-15T12:40:51.519 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:40:51.519 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-05-15T12:40:51.519 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:40:51.519 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-05-15T12:40:51.519 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:40:51.519 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 615 k 2024-05-15T12:40:51.519 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 2.1 M 2024-05-15T12:40:51.519 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-05-15T12:40:52.905 INFO:teuthology.orchestra.run.smithi178.stdout:python3-cephfs-debuginfo-19.0.0-1882.gae248125. 444 kB/s | 615 kB 00:01 2024-05-15T12:40:52.905 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:40:52.906 INFO:teuthology.orchestra.run.smithi178.stdout:Total 443 kB/s | 615 kB 00:01 2024-05-15T12:40:52.906 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-05-15T12:40:52.910 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-05-15T12:40:52.910 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-05-15T12:40:52.950 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-05-15T12:40:52.951 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-05-15T12:40:53.116 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-05-15T12:40:53.220 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1882.gae248125.el9 1/1 2024-05-15T12:40:53.624 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1882.gae248125.el9 1/1 2024-05-15T12:40:53.846 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1882.gae248125.el9 1/1 2024-05-15T12:40:53.846 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:40:53.846 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-05-15T12:40:53.846 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cephfs-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:40:53.846 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:40:53.846 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-05-15T12:40:54.063 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install python3-rados-debuginfo 2024-05-15T12:40:54.641 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:09:29 ago on Wed 15 May 2024 12:31:25 PM UTC. 2024-05-15T12:40:54.763 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-05-15T12:40:54.764 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:40:54.764 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-05-15T12:40:54.764 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:40:54.764 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-05-15T12:40:54.764 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 769 k 2024-05-15T12:40:54.764 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:40:54.764 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-05-15T12:40:54.764 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:40:54.764 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-05-15T12:40:54.764 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:40:54.765 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 769 k 2024-05-15T12:40:54.765 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 2.7 M 2024-05-15T12:40:54.765 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-05-15T12:40:55.432 INFO:teuthology.orchestra.run.smithi178.stdout:python3-rados-debuginfo-19.0.0-1882.gae248125.e 1.1 MB/s | 769 kB 00:00 2024-05-15T12:40:55.433 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:40:55.433 INFO:teuthology.orchestra.run.smithi178.stdout:Total 1.1 MB/s | 769 kB 00:00 2024-05-15T12:40:55.433 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-05-15T12:40:55.437 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-05-15T12:40:55.437 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-05-15T12:40:55.476 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-05-15T12:40:55.476 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-05-15T12:40:55.656 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-05-15T12:40:55.747 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1882.gae248125.el9. 1/1 2024-05-15T12:40:56.065 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1882.gae248125.el9. 1/1 2024-05-15T12:40:56.287 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1882.gae248125.el9. 1/1 2024-05-15T12:40:56.287 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:40:56.287 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-05-15T12:40:56.287 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rados-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:40:56.287 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:40:56.287 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-05-15T12:40:56.504 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install python3-rbd-debuginfo 2024-05-15T12:40:57.082 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:09:32 ago on Wed 15 May 2024 12:31:25 PM UTC. 2024-05-15T12:40:57.204 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-05-15T12:40:57.205 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:40:57.205 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-05-15T12:40:57.205 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:40:57.205 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-05-15T12:40:57.205 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 1.1 M 2024-05-15T12:40:57.205 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:40:57.205 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-05-15T12:40:57.205 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:40:57.205 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-05-15T12:40:57.205 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:40:57.206 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 1.1 M 2024-05-15T12:40:57.206 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 4.3 M 2024-05-15T12:40:57.206 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-05-15T12:40:57.760 INFO:teuthology.orchestra.run.smithi178.stdout:python3-rbd-debuginfo-19.0.0-1882.gae248125.el9 2.0 MB/s | 1.1 MB 00:00 2024-05-15T12:40:57.760 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:40:57.761 INFO:teuthology.orchestra.run.smithi178.stdout:Total 2.0 MB/s | 1.1 MB 00:00 2024-05-15T12:40:57.761 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-05-15T12:40:57.764 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-05-15T12:40:57.764 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-05-15T12:40:57.805 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-05-15T12:40:57.805 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-05-15T12:40:58.028 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-05-15T12:40:58.125 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1882.gae248125.el9.x8 1/1 2024-05-15T12:40:58.514 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1882.gae248125.el9.x8 1/1 2024-05-15T12:40:58.754 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1882.gae248125.el9.x8 1/1 2024-05-15T12:40:58.754 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:40:58.754 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-05-15T12:40:58.754 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rbd-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:40:58.754 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:40:58.754 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-05-15T12:40:58.986 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install python3-rgw-debuginfo 2024-05-15T12:40:59.564 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:09:34 ago on Wed 15 May 2024 12:31:25 PM UTC. 2024-05-15T12:40:59.685 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-05-15T12:40:59.686 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:40:59.686 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-05-15T12:40:59.686 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:40:59.686 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-05-15T12:40:59.686 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 337 k 2024-05-15T12:40:59.686 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:40:59.686 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-05-15T12:40:59.686 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:40:59.686 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-05-15T12:40:59.687 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:40:59.687 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 337 k 2024-05-15T12:40:59.687 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 1.0 M 2024-05-15T12:40:59.687 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-05-15T12:41:00.146 INFO:teuthology.orchestra.run.smithi178.stdout:python3-rgw-debuginfo-19.0.0-1882.gae248125.el9 735 kB/s | 337 kB 00:00 2024-05-15T12:41:00.146 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:41:00.147 INFO:teuthology.orchestra.run.smithi178.stdout:Total 733 kB/s | 337 kB 00:00 2024-05-15T12:41:00.147 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-05-15T12:41:00.150 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-05-15T12:41:00.150 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-05-15T12:41:00.186 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-05-15T12:41:00.187 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-05-15T12:41:00.327 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-05-15T12:41:00.444 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1882.gae248125.el9.x8 1/1 2024-05-15T12:41:00.799 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1882.gae248125.el9.x8 1/1 2024-05-15T12:41:01.027 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1882.gae248125.el9.x8 1/1 2024-05-15T12:41:01.028 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:41:01.028 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-05-15T12:41:01.028 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rgw-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:41:01.028 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:41:01.028 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-05-15T12:41:01.228 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install rbd-fuse-debuginfo 2024-05-15T12:41:01.800 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:09:36 ago on Wed 15 May 2024 12:31:25 PM UTC. 2024-05-15T12:41:01.926 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-05-15T12:41:01.927 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:41:01.927 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-05-15T12:41:01.927 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:41:01.927 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-05-15T12:41:01.927 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 517 k 2024-05-15T12:41:01.927 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:41:01.927 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-05-15T12:41:01.927 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:41:01.927 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-05-15T12:41:01.927 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:41:01.927 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 517 k 2024-05-15T12:41:01.927 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 2.6 M 2024-05-15T12:41:01.927 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-05-15T12:41:02.441 INFO:teuthology.orchestra.run.smithi178.stdout:rbd-fuse-debuginfo-19.0.0-1882.gae248125.el9.x8 1.0 MB/s | 517 kB 00:00 2024-05-15T12:41:02.441 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:41:02.442 INFO:teuthology.orchestra.run.smithi178.stdout:Total 1.0 MB/s | 517 kB 00:00 2024-05-15T12:41:02.442 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-05-15T12:41:02.445 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-05-15T12:41:02.445 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-05-15T12:41:02.484 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-05-15T12:41:02.484 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-05-15T12:41:02.641 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-05-15T12:41:02.740 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:41:03.111 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:41:03.328 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:41:03.328 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:41:03.328 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-05-15T12:41:03.328 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-fuse-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:41:03.328 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:41:03.328 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-05-15T12:41:03.525 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install rbd-mirror-debuginfo 2024-05-15T12:41:04.103 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:09:39 ago on Wed 15 May 2024 12:31:25 PM UTC. 2024-05-15T12:41:04.227 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-05-15T12:41:04.228 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:41:04.228 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-05-15T12:41:04.228 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:41:04.228 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-05-15T12:41:04.228 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 52 M 2024-05-15T12:41:04.228 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:41:04.228 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-05-15T12:41:04.228 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:41:04.228 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-05-15T12:41:04.228 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:41:04.229 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 52 M 2024-05-15T12:41:04.229 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 298 M 2024-05-15T12:41:04.229 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-05-15T12:41:07.302 INFO:teuthology.orchestra.run.smithi178.stdout:rbd-mirror-debuginfo-19.0.0-1882.gae248125.el9. 17 MB/s | 52 MB 00:03 2024-05-15T12:41:07.302 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:41:07.303 INFO:teuthology.orchestra.run.smithi178.stdout:Total 17 MB/s | 52 MB 00:03 2024-05-15T12:41:07.303 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-05-15T12:41:07.307 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-05-15T12:41:07.307 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-05-15T12:41:07.644 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-05-15T12:41:07.645 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-05-15T12:41:13.566 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-05-15T12:41:13.732 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1882.gae248125.el9.x86 1/1 2024-05-15T12:41:15.829 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1882.gae248125.el9.x86 1/1 2024-05-15T12:41:16.051 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1882.gae248125.el9.x86 1/1 2024-05-15T12:41:16.051 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:41:16.051 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-05-15T12:41:16.051 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-mirror-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:41:16.051 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:41:16.051 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-05-15T12:41:16.241 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install rbd-nbd-debuginfo 2024-05-15T12:41:16.819 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:09:51 ago on Wed 15 May 2024 12:31:25 PM UTC. 2024-05-15T12:41:16.942 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-05-15T12:41:16.943 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:41:16.943 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-05-15T12:41:16.943 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:41:16.943 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-05-15T12:41:16.943 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 1.3 M 2024-05-15T12:41:16.943 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:41:16.944 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-05-15T12:41:16.944 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-05-15T12:41:16.944 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-05-15T12:41:16.944 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:41:16.944 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 1.3 M 2024-05-15T12:41:16.944 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 5.7 M 2024-05-15T12:41:16.944 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-05-15T12:41:17.437 INFO:teuthology.orchestra.run.smithi178.stdout:rbd-nbd-debuginfo-19.0.0-1882.gae248125.el9.x86 2.6 MB/s | 1.3 MB 00:00 2024-05-15T12:41:17.437 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:41:17.438 INFO:teuthology.orchestra.run.smithi178.stdout:Total 2.6 MB/s | 1.3 MB 00:00 2024-05-15T12:41:17.438 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-05-15T12:41:17.441 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-05-15T12:41:17.441 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-05-15T12:41:17.484 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-05-15T12:41:17.485 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-05-15T12:41:17.726 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-05-15T12:41:17.835 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:41:18.243 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:41:18.474 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:41:18.474 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:41:18.474 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-05-15T12:41:18.474 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-nbd-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:41:18.474 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:41:18.474 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-05-15T12:41:18.672 DEBUG:teuthology.parallel:result is None 2024-05-15T12:41:18.672 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=ae2481253f9b2fa3451954e615eedb67ebdd5314 2024-05-15T12:41:18.888 DEBUG:teuthology.orchestra.run.smithi117:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-15T12:41:18.914 INFO:teuthology.orchestra.run.smithi117.stdout:19.0.0-1882.gae248125.el9 2024-05-15T12:41:18.915 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1882.gae248125.el9 2024-05-15T12:41:18.915 INFO:teuthology.task.install:The correct ceph version 19.0.0-1882.gae248125 is installed. 2024-05-15T12:41:18.917 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=ae2481253f9b2fa3451954e615eedb67ebdd5314 2024-05-15T12:41:19.133 DEBUG:teuthology.orchestra.run.smithi178:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-15T12:41:19.154 INFO:teuthology.orchestra.run.smithi178.stdout:19.0.0-1882.gae248125.el9 2024-05-15T12:41:19.154 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1882.gae248125.el9 2024-05-15T12:41:19.155 INFO:teuthology.task.install:The correct ceph version 19.0.0-1882.gae248125 is installed. 2024-05-15T12:41:19.156 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-05-15T12:41:19.156 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-05-15T12:41:19.156 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-15T12:41:19.188 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-05-15T12:41:19.188 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-15T12:41:19.220 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-05-15T12:41:19.220 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-05-15T12:41:19.220 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/usr/bin/daemon-helper 2024-05-15T12:41:19.257 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-15T12:41:19.325 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-05-15T12:41:19.325 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/usr/bin/daemon-helper 2024-05-15T12:41:19.349 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-15T12:41:19.414 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-05-15T12:41:19.414 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-05-15T12:41:19.414 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-15T12:41:19.443 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-15T12:41:19.509 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-05-15T12:41:19.509 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-15T12:41:19.534 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-15T12:41:19.598 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-05-15T12:41:19.598 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-05-15T12:41:19.598 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/usr/bin/stdin-killer 2024-05-15T12:41:19.624 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-15T12:41:19.690 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-05-15T12:41:19.690 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/usr/bin/stdin-killer 2024-05-15T12:41:19.715 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-15T12:41:19.779 INFO:teuthology.run_tasks:Running task ceph... 2024-05-15T12:41:19.855 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-05-15T12:41:19.855 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 777 /var/log/ceph 2024-05-15T12:41:19.858 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod 777 /var/log/ceph 2024-05-15T12:41:19.882 INFO:tasks.ceph:Disabling ceph logrotate... 2024-05-15T12:41:19.882 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f -- /etc/logrotate.d/ceph 2024-05-15T12:41:19.925 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -f -- /etc/logrotate.d/ceph 2024-05-15T12:41:19.949 INFO:tasks.ceph:Creating extra log directories... 2024-05-15T12:41:19.949 DEBUG:teuthology.orchestra.run.smithi117:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-05-15T12:41:19.991 DEBUG:teuthology.orchestra.run.smithi178:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-05-15T12:41:20.025 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-05-15T12:41:20.025 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-05-15T12:41:20.025 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707111', 'branch': 'wip-osd-ec-partial-reads', 'description': 'rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7707111', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-osd-ec-partial-reads'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'sha1': 'ae2481253f9b2fa3451954e615eedb67ebdd5314', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': 'ae2481253f9b2fa3451954e615eedb67ebdd5314'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-osd-ec-partial-reads', 'sha1': 'ae2481253f9b2fa3451954e615eedb67ebdd5314'}}, 'owner': 'scheduled_rzarzynski@teuthology', 'priority': 85, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 8148, 'sha1': 'ae2481253f9b2fa3451954e615eedb67ebdd5314', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-osd-ec-partial-reads', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_ae2481253f9b2fa3451954e615eedb67ebdd5314/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'ae2481253f9b2fa3451954e615eedb67ebdd5314', 'targets': {'smithi117.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIsXkaK4XQPzB8+ZJdXcA3a5GBSnAV9wzg7BG9USRcj7bw0CjWp7FrSJylm7crDcKiGM5FFlopish4Bjir3K9qc=', 'smithi178.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPHTLb3nCf+LZPtkT+i7iba/GITUHDkKt0Jf053EyLYISLrDwZ2shMTtn0NUH8ax5kcE9uVIvY6w5Ahszapyv4w='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '1ae7ad82388e92a475afff437d49054826c019a1', 'timestamp': '2024-05-15_11:53:37', 'tube': 'smithi', 'user': 'rzarzynski', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-05-15T12:41:20.026 DEBUG:teuthology.orchestra.run.smithi117:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-05-15T12:41:20.057 DEBUG:teuthology.orchestra.run.smithi178:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-05-15T12:41:20.072 DEBUG:teuthology.orchestra.run.smithi117:> sudo install -d -m0777 -- /var/run/ceph 2024-05-15T12:41:20.113 DEBUG:teuthology.orchestra.run.smithi178:> sudo install -d -m0777 -- /var/run/ceph 2024-05-15T12:41:20.138 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-05-15T12:41:20.138 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/scratch_devs of=/dev/stdout 2024-05-15T12:41:20.193 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-15T12:41:20.193 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_1 2024-05-15T12:41:20.247 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-15T12:41:20.248 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-15T12:41:20.248 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-05-15T12:41:20.248 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-15T12:41:20.248 INFO:teuthology.orchestra.run.smithi117.stdout:Context: system_u:object_r:device_t:s0 2024-05-15T12:41:20.248 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-05-15 12:33:14.113062673 +0000 2024-05-15T12:41:20.248 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-05-15 12:30:25.743005368 +0000 2024-05-15T12:41:20.248 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-05-15 12:30:25.743005368 +0000 2024-05-15T12:41:20.248 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: 2024-05-15 12:30:25.743005368 +0000 2024-05-15T12:41:20.248 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-15T12:41:20.310 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-05-15T12:41:20.310 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-05-15T12:41:20.310 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000203525 s, 2.5 MB/s 2024-05-15T12:41:20.311 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-15T12:41:20.368 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_2 2024-05-15T12:41:20.423 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-15T12:41:20.423 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-15T12:41:20.423 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-05-15T12:41:20.424 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-15T12:41:20.424 INFO:teuthology.orchestra.run.smithi117.stdout:Context: system_u:object_r:device_t:s0 2024-05-15T12:41:20.424 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-05-15 12:33:14.114062655 +0000 2024-05-15T12:41:20.424 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-05-15 12:30:26.100998762 +0000 2024-05-15T12:41:20.424 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-05-15 12:30:26.100998762 +0000 2024-05-15T12:41:20.424 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: 2024-05-15 12:30:26.100998762 +0000 2024-05-15T12:41:20.424 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-15T12:41:20.487 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-05-15T12:41:20.487 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-05-15T12:41:20.487 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000217645 s, 2.4 MB/s 2024-05-15T12:41:20.489 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-15T12:41:20.544 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_3 2024-05-15T12:41:20.600 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-15T12:41:20.600 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-15T12:41:20.600 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-05-15T12:41:20.600 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-15T12:41:20.600 INFO:teuthology.orchestra.run.smithi117.stdout:Context: system_u:object_r:device_t:s0 2024-05-15T12:41:20.600 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-05-15 12:33:14.114062655 +0000 2024-05-15T12:41:20.600 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-05-15 12:30:26.456992192 +0000 2024-05-15T12:41:20.600 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-05-15 12:30:26.456992192 +0000 2024-05-15T12:41:20.600 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: 2024-05-15 12:30:26.456992192 +0000 2024-05-15T12:41:20.601 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-15T12:41:20.663 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-05-15T12:41:20.663 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-05-15T12:41:20.663 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000200921 s, 2.5 MB/s 2024-05-15T12:41:20.664 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-15T12:41:20.720 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_4 2024-05-15T12:41:20.776 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-15T12:41:20.776 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-15T12:41:20.776 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-05-15T12:41:20.776 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-15T12:41:20.776 INFO:teuthology.orchestra.run.smithi117.stdout:Context: system_u:object_r:device_t:s0 2024-05-15T12:41:20.776 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-05-15 12:33:14.115062636 +0000 2024-05-15T12:41:20.776 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-05-15 12:30:26.789986046 +0000 2024-05-15T12:41:20.776 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-05-15 12:30:26.789986046 +0000 2024-05-15T12:41:20.776 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: 2024-05-15 12:30:26.789986046 +0000 2024-05-15T12:41:20.776 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-15T12:41:20.838 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-05-15T12:41:20.838 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-05-15T12:41:20.838 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000192431 s, 2.7 MB/s 2024-05-15T12:41:20.840 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-15T12:41:20.896 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-15T12:41:20.896 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-05-15T12:41:20.896 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/scratch_devs of=/dev/stdout 2024-05-15T12:41:20.915 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-15T12:41:20.915 DEBUG:teuthology.orchestra.run.smithi178:> stat /dev/vg_nvme/lv_1 2024-05-15T12:41:20.973 INFO:teuthology.orchestra.run.smithi178.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-15T12:41:20.973 INFO:teuthology.orchestra.run.smithi178.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-15T12:41:20.973 INFO:teuthology.orchestra.run.smithi178.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-05-15T12:41:20.973 INFO:teuthology.orchestra.run.smithi178.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-15T12:41:20.973 INFO:teuthology.orchestra.run.smithi178.stdout:Context: system_u:object_r:device_t:s0 2024-05-15T12:41:20.974 INFO:teuthology.orchestra.run.smithi178.stdout:Access: 2024-05-15 12:33:44.197462161 +0000 2024-05-15T12:41:20.974 INFO:teuthology.orchestra.run.smithi178.stdout:Modify: 2024-05-15 12:30:25.730514932 +0000 2024-05-15T12:41:20.974 INFO:teuthology.orchestra.run.smithi178.stdout:Change: 2024-05-15 12:30:25.730514932 +0000 2024-05-15T12:41:20.974 INFO:teuthology.orchestra.run.smithi178.stdout: Birth: 2024-05-15 12:30:25.730514932 +0000 2024-05-15T12:41:20.974 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-15T12:41:21.041 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records in 2024-05-15T12:41:21.042 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records out 2024-05-15T12:41:21.042 INFO:teuthology.orchestra.run.smithi178.stderr:512 bytes copied, 0.000100269 s, 5.1 MB/s 2024-05-15T12:41:21.043 DEBUG:teuthology.orchestra.run.smithi178:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-15T12:41:21.102 DEBUG:teuthology.orchestra.run.smithi178:> stat /dev/vg_nvme/lv_2 2024-05-15T12:41:21.160 INFO:teuthology.orchestra.run.smithi178.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-15T12:41:21.160 INFO:teuthology.orchestra.run.smithi178.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-15T12:41:21.160 INFO:teuthology.orchestra.run.smithi178.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-05-15T12:41:21.160 INFO:teuthology.orchestra.run.smithi178.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-15T12:41:21.160 INFO:teuthology.orchestra.run.smithi178.stdout:Context: system_u:object_r:device_t:s0 2024-05-15T12:41:21.160 INFO:teuthology.orchestra.run.smithi178.stdout:Access: 2024-05-15 12:33:44.198462143 +0000 2024-05-15T12:41:21.160 INFO:teuthology.orchestra.run.smithi178.stdout:Modify: 2024-05-15 12:30:26.066508883 +0000 2024-05-15T12:41:21.160 INFO:teuthology.orchestra.run.smithi178.stdout:Change: 2024-05-15 12:30:26.066508883 +0000 2024-05-15T12:41:21.160 INFO:teuthology.orchestra.run.smithi178.stdout: Birth: 2024-05-15 12:30:26.066508883 +0000 2024-05-15T12:41:21.161 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-15T12:41:21.226 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records in 2024-05-15T12:41:21.226 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records out 2024-05-15T12:41:21.226 INFO:teuthology.orchestra.run.smithi178.stderr:512 bytes copied, 0.000114349 s, 4.5 MB/s 2024-05-15T12:41:21.228 DEBUG:teuthology.orchestra.run.smithi178:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-15T12:41:21.286 DEBUG:teuthology.orchestra.run.smithi178:> stat /dev/vg_nvme/lv_3 2024-05-15T12:41:21.342 INFO:teuthology.orchestra.run.smithi178.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-15T12:41:21.342 INFO:teuthology.orchestra.run.smithi178.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-15T12:41:21.343 INFO:teuthology.orchestra.run.smithi178.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-05-15T12:41:21.343 INFO:teuthology.orchestra.run.smithi178.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-15T12:41:21.343 INFO:teuthology.orchestra.run.smithi178.stdout:Context: system_u:object_r:device_t:s0 2024-05-15T12:41:21.343 INFO:teuthology.orchestra.run.smithi178.stdout:Access: 2024-05-15 12:33:44.198462143 +0000 2024-05-15T12:41:21.343 INFO:teuthology.orchestra.run.smithi178.stdout:Modify: 2024-05-15 12:30:26.390503050 +0000 2024-05-15T12:41:21.343 INFO:teuthology.orchestra.run.smithi178.stdout:Change: 2024-05-15 12:30:26.390503050 +0000 2024-05-15T12:41:21.343 INFO:teuthology.orchestra.run.smithi178.stdout: Birth: 2024-05-15 12:30:26.390503050 +0000 2024-05-15T12:41:21.343 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-15T12:41:21.408 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records in 2024-05-15T12:41:21.408 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records out 2024-05-15T12:41:21.408 INFO:teuthology.orchestra.run.smithi178.stderr:512 bytes copied, 0.000131231 s, 3.9 MB/s 2024-05-15T12:41:21.409 DEBUG:teuthology.orchestra.run.smithi178:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-15T12:41:21.467 DEBUG:teuthology.orchestra.run.smithi178:> stat /dev/vg_nvme/lv_4 2024-05-15T12:41:21.523 INFO:teuthology.orchestra.run.smithi178.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-15T12:41:21.523 INFO:teuthology.orchestra.run.smithi178.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-15T12:41:21.523 INFO:teuthology.orchestra.run.smithi178.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-05-15T12:41:21.523 INFO:teuthology.orchestra.run.smithi178.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-15T12:41:21.523 INFO:teuthology.orchestra.run.smithi178.stdout:Context: system_u:object_r:device_t:s0 2024-05-15T12:41:21.523 INFO:teuthology.orchestra.run.smithi178.stdout:Access: 2024-05-15 12:33:44.198462143 +0000 2024-05-15T12:41:21.523 INFO:teuthology.orchestra.run.smithi178.stdout:Modify: 2024-05-15 12:30:26.720497108 +0000 2024-05-15T12:41:21.523 INFO:teuthology.orchestra.run.smithi178.stdout:Change: 2024-05-15 12:30:26.720497108 +0000 2024-05-15T12:41:21.523 INFO:teuthology.orchestra.run.smithi178.stdout: Birth: 2024-05-15 12:30:26.720497108 +0000 2024-05-15T12:41:21.523 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-15T12:41:21.587 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records in 2024-05-15T12:41:21.587 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records out 2024-05-15T12:41:21.587 INFO:teuthology.orchestra.run.smithi178.stderr:512 bytes copied, 8.0018e-05 s, 6.4 MB/s 2024-05-15T12:41:21.588 DEBUG:teuthology.orchestra.run.smithi178:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-15T12:41:21.645 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-15T12:41:21.645 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi117.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi178.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-05-15T12:41:21.645 INFO:tasks.ceph:Generating config... 2024-05-15T12:41:21.646 INFO:tasks.ceph:[client] debug monc = 20 2024-05-15T12:41:21.646 INFO:tasks.ceph:[client] debug ms = 1 2024-05-15T12:41:21.646 INFO:tasks.ceph:[client] debug objecter = 20 2024-05-15T12:41:21.647 INFO:tasks.ceph:[client] debug rados = 20 2024-05-15T12:41:21.647 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-05-15T12:41:21.647 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-05-15T12:41:21.647 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-05-15T12:41:21.647 INFO:tasks.ceph:[global] ms type = async 2024-05-15T12:41:21.647 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-05-15T12:41:21.647 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-05-15T12:41:21.647 INFO:tasks.ceph:[mgr] debug ms = 1 2024-05-15T12:41:21.647 INFO:tasks.ceph:[mon] debug mon = 20 2024-05-15T12:41:21.647 INFO:tasks.ceph:[mon] debug ms = 1 2024-05-15T12:41:21.647 INFO:tasks.ceph:[mon] debug paxos = 20 2024-05-15T12:41:21.647 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-05-15T12:41:21.647 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-05-15T12:41:21.647 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-05-15T12:41:21.648 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-05-15T12:41:21.648 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-05-15T12:41:21.648 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-05-15T12:41:21.648 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-05-15T12:41:21.648 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-05-15T12:41:21.648 INFO:tasks.ceph:[osd] bdev async discard = True 2024-05-15T12:41:21.648 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-05-15T12:41:21.648 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-05-15T12:41:21.648 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-05-15T12:41:21.648 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-05-15T12:41:21.648 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-05-15T12:41:21.648 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-05-15T12:41:21.648 INFO:tasks.ceph:[osd] debug monc = 20 2024-05-15T12:41:21.648 INFO:tasks.ceph:[osd] debug ms = 1 2024-05-15T12:41:21.648 INFO:tasks.ceph:[osd] debug osd = 20 2024-05-15T12:41:21.648 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-05-15T12:41:21.649 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-05-15T12:41:21.649 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-05-15T12:41:21.649 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-05-15T12:41:21.649 INFO:tasks.ceph:[osd] osd class default list = * 2024-05-15T12:41:21.649 INFO:tasks.ceph:[osd] osd class load list = * 2024-05-15T12:41:21.649 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-05-15T12:41:21.649 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-05-15T12:41:21.649 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-05-15T12:41:21.649 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-05-15T12:41:21.649 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-05-15T12:41:21.649 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-05-15T12:41:21.649 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-05-15T12:41:21.649 INFO:tasks.ceph:[osd] osd memory target = 1610612736 2024-05-15T12:41:21.649 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-05-15T12:41:21.649 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-05-15T12:41:21.650 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-05-15T12:41:21.650 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-05-15T12:41:21.650 INFO:tasks.ceph:Setting up mon.a... 2024-05-15T12:41:21.650 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-05-15T12:41:21.697 INFO:teuthology.orchestra.run.smithi117.stdout:creating /etc/ceph/ceph.keyring 2024-05-15T12:41:21.700 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-05-15T12:41:21.744 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-05-15T12:41:21.769 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.117'), ('mon.c', '[v2:172.21.15.117:3301,v1:172.21.15.117:6790]'), ('mon.b', '172.21.15.178')] 2024-05-15T12:41:21.769 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.117,[v2:172.21.15.117:3301,v1:172.21.15.117:6790],172.21.15.178', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-05-15T12:41:21.770 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-05-15T12:41:21.770 DEBUG:teuthology.orchestra.run.smithi117:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-05-15T12:41:21.824 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.117 --addv c '[v2:172.21.15.117:3301,v1:172.21.15.117:6790]' --add b 172.21.15.178 --print /home/ubuntu/cephtest/ceph.monmap 2024-05-15T12:41:21.910 INFO:teuthology.orchestra.run.smithi117.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-05-15T12:41:21.911 INFO:teuthology.orchestra.run.smithi117.stdout:monmaptool: generated fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:41:21.911 INFO:teuthology.orchestra.run.smithi117.stdout:setting min_mon_release = quincy 2024-05-15T12:41:21.911 INFO:teuthology.orchestra.run.smithi117.stdout:epoch 0 2024-05-15T12:41:21.911 INFO:teuthology.orchestra.run.smithi117.stdout:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:41:21.911 INFO:teuthology.orchestra.run.smithi117.stdout:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:41:21.911 INFO:teuthology.orchestra.run.smithi117.stdout:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:41:21.911 INFO:teuthology.orchestra.run.smithi117.stdout:min_mon_release 17 (quincy) 2024-05-15T12:41:21.911 INFO:teuthology.orchestra.run.smithi117.stdout:election_strategy: 1 2024-05-15T12:41:21.911 INFO:teuthology.orchestra.run.smithi117.stdout:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:41:21.911 INFO:teuthology.orchestra.run.smithi117.stdout:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:41:21.911 INFO:teuthology.orchestra.run.smithi117.stdout:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:41:21.911 INFO:teuthology.orchestra.run.smithi117.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-05-15T12:41:21.913 DEBUG:teuthology.orchestra.run.smithi117:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-05-15T12:41:21.928 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 8817f6cf-d933-4825-b972-575ef59f5ba9... 2024-05-15T12:41:21.928 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-05-15T12:41:21.970 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-05-15T12:41:22.010 INFO:teuthology.orchestra.run.smithi117.stdout:[global] 2024-05-15T12:41:22.010 INFO:teuthology.orchestra.run.smithi117.stdout: chdir = "" 2024-05-15T12:41:22.010 INFO:teuthology.orchestra.run.smithi117.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-05-15T12:41:22.010 INFO:teuthology.orchestra.run.smithi117.stdout: auth supported = cephx 2024-05-15T12:41:22.010 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:41:22.010 INFO:teuthology.orchestra.run.smithi117.stdout: filestore xattr use omap = true 2024-05-15T12:41:22.010 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:41:22.010 INFO:teuthology.orchestra.run.smithi117.stdout: mon clock drift allowed = 1.000 2024-05-15T12:41:22.010 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:41:22.010 INFO:teuthology.orchestra.run.smithi117.stdout: osd crush chooseleaf type = 0 2024-05-15T12:41:22.010 INFO:teuthology.orchestra.run.smithi117.stdout: auth debug = true 2024-05-15T12:41:22.011 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:41:22.011 INFO:teuthology.orchestra.run.smithi117.stdout: ms die on old message = true 2024-05-15T12:41:22.011 INFO:teuthology.orchestra.run.smithi117.stdout: ms die on bug = true 2024-05-15T12:41:22.011 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:41:22.011 INFO:teuthology.orchestra.run.smithi117.stdout: mon max pg per osd = 10000 # >= luminous 2024-05-15T12:41:22.011 INFO:teuthology.orchestra.run.smithi117.stdout: mon pg warn max object skew = 0 2024-05-15T12:41:22.011 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:41:22.011 INFO:teuthology.orchestra.run.smithi117.stdout: # disable pg_autoscaler by default for new pools 2024-05-15T12:41:22.011 INFO:teuthology.orchestra.run.smithi117.stdout: osd_pool_default_pg_autoscale_mode = off 2024-05-15T12:41:22.011 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:41:22.011 INFO:teuthology.orchestra.run.smithi117.stdout: osd pool default size = 2 2024-05-15T12:41:22.011 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:41:22.011 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd allow primary affinity = true 2024-05-15T12:41:22.011 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd allow pg remap = true 2024-05-15T12:41:22.011 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on legacy crush tunables = false 2024-05-15T12:41:22.012 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on crush straw calc version zero = false 2024-05-15T12:41:22.012 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on no sortbitwise = false 2024-05-15T12:41:22.012 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on osd down out interval zero = false 2024-05-15T12:41:22.012 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on too few osds = false 2024-05-15T12:41:22.012 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-05-15T12:41:22.012 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_pool_no_redundancy = false 2024-05-15T12:41:22.012 INFO:teuthology.orchestra.run.smithi117.stdout: mon_allow_pool_size_one = true 2024-05-15T12:41:22.012 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:41:22.012 INFO:teuthology.orchestra.run.smithi117.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-05-15T12:41:22.012 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:41:22.012 INFO:teuthology.orchestra.run.smithi117.stdout: osd default data pool replay window = 5 2024-05-15T12:41:22.012 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:41:22.012 INFO:teuthology.orchestra.run.smithi117.stdout: mon allow pool delete = true 2024-05-15T12:41:22.012 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:41:22.012 INFO:teuthology.orchestra.run.smithi117.stdout: mon cluster log file level = debug 2024-05-15T12:41:22.012 INFO:teuthology.orchestra.run.smithi117.stdout: debug asserts on shutdown = true 2024-05-15T12:41:22.013 INFO:teuthology.orchestra.run.smithi117.stdout: mon health detail to clog = false 2024-05-15T12:41:22.013 INFO:teuthology.orchestra.run.smithi117.stdout: mon host = "172.21.15.117,[v2:172.21.15.117:3301,v1:172.21.15.117:6790],172.21.15.178" 2024-05-15T12:41:22.013 INFO:teuthology.orchestra.run.smithi117.stdout: mon client directed command retry = 5 2024-05-15T12:41:22.013 INFO:teuthology.orchestra.run.smithi117.stdout: mon election default strategy = 1 2024-05-15T12:41:22.013 INFO:teuthology.orchestra.run.smithi117.stdout: ms inject socket failures = 5000 2024-05-15T12:41:22.013 INFO:teuthology.orchestra.run.smithi117.stdout: ms type = async 2024-05-15T12:41:22.013 INFO:teuthology.orchestra.run.smithi117.stdout: osd heartbeat grace = 80 2024-05-15T12:41:22.013 INFO:teuthology.orchestra.run.smithi117.stdout: fsid = 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:41:22.013 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:41:22.013 INFO:teuthology.orchestra.run.smithi117.stdout:[osd] 2024-05-15T12:41:22.013 INFO:teuthology.orchestra.run.smithi117.stdout: osd journal size = 100 2024-05-15T12:41:22.013 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:41:22.013 INFO:teuthology.orchestra.run.smithi117.stdout: osd scrub load threshold = 5.0 2024-05-15T12:41:22.013 INFO:teuthology.orchestra.run.smithi117.stdout: osd scrub max interval = 600 2024-05-15T12:41:22.013 INFO:teuthology.orchestra.run.smithi117.stdout: osd mclock profile = high_recovery_ops 2024-05-15T12:41:22.014 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:41:22.014 INFO:teuthology.orchestra.run.smithi117.stdout: osd recover clone overlap = true 2024-05-15T12:41:22.014 INFO:teuthology.orchestra.run.smithi117.stdout: osd recovery max chunk = 1048576 2024-05-15T12:41:22.014 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:41:22.014 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug shutdown = true 2024-05-15T12:41:22.014 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug op order = true 2024-05-15T12:41:22.014 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug verify stray on activate = true 2024-05-15T12:41:22.014 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug trim objects = true 2024-05-15T12:41:22.014 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:41:22.014 INFO:teuthology.orchestra.run.smithi117.stdout: osd open classes on start = true 2024-05-15T12:41:22.014 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug pg log writeout = true 2024-05-15T12:41:22.014 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:41:22.014 INFO:teuthology.orchestra.run.smithi117.stdout: osd deep scrub update digest min age = 30 2024-05-15T12:41:22.014 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:41:22.015 INFO:teuthology.orchestra.run.smithi117.stdout: osd map max advance = 10 2024-05-15T12:41:22.015 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:41:22.015 INFO:teuthology.orchestra.run.smithi117.stdout: journal zero on create = true 2024-05-15T12:41:22.015 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:41:22.015 INFO:teuthology.orchestra.run.smithi117.stdout: filestore ondisk finisher threads = 3 2024-05-15T12:41:22.015 INFO:teuthology.orchestra.run.smithi117.stdout: filestore apply finisher threads = 3 2024-05-15T12:41:22.015 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:41:22.015 INFO:teuthology.orchestra.run.smithi117.stdout: bdev debug aio = true 2024-05-15T12:41:22.015 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug misdirected ops = true 2024-05-15T12:41:22.015 INFO:teuthology.orchestra.run.smithi117.stdout: bdev async discard = True 2024-05-15T12:41:22.015 INFO:teuthology.orchestra.run.smithi117.stdout: bdev enable discard = True 2024-05-15T12:41:22.016 INFO:teuthology.orchestra.run.smithi117.stdout: bluestore block size = 96636764160 2024-05-15T12:41:22.016 INFO:teuthology.orchestra.run.smithi117.stdout: bluestore fsck on mount = True 2024-05-15T12:41:22.016 INFO:teuthology.orchestra.run.smithi117.stdout: bluestore zero block detection = True 2024-05-15T12:41:22.016 INFO:teuthology.orchestra.run.smithi117.stdout: debug bluefs = 1 2024-05-15T12:41:22.016 INFO:teuthology.orchestra.run.smithi117.stdout: debug bluestore = 1 2024-05-15T12:41:22.016 INFO:teuthology.orchestra.run.smithi117.stdout: debug monc = 20 2024-05-15T12:41:22.016 INFO:teuthology.orchestra.run.smithi117.stdout: debug ms = 1 2024-05-15T12:41:22.016 INFO:teuthology.orchestra.run.smithi117.stdout: debug osd = 20 2024-05-15T12:41:22.016 INFO:teuthology.orchestra.run.smithi117.stdout: debug rocksdb = 10 2024-05-15T12:41:22.016 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd backfillfull_ratio = 0.85 2024-05-15T12:41:22.016 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd full ratio = 0.9 2024-05-15T12:41:22.016 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd nearfull ratio = 0.8 2024-05-15T12:41:22.016 INFO:teuthology.orchestra.run.smithi117.stdout: osd class default list = * 2024-05-15T12:41:22.016 INFO:teuthology.orchestra.run.smithi117.stdout: osd class load list = * 2024-05-15T12:41:22.017 INFO:teuthology.orchestra.run.smithi117.stdout: osd client watch timeout = 120 2024-05-15T12:41:22.017 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug verify cached snaps = True 2024-05-15T12:41:22.017 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug verify missing on start = True 2024-05-15T12:41:22.017 INFO:teuthology.orchestra.run.smithi117.stdout: osd failsafe full ratio = 0.95 2024-05-15T12:41:22.017 INFO:teuthology.orchestra.run.smithi117.stdout: osd fast shutdown = False 2024-05-15T12:41:22.017 INFO:teuthology.orchestra.run.smithi117.stdout: osd mclock override recovery settings = True 2024-05-15T12:41:22.017 INFO:teuthology.orchestra.run.smithi117.stdout: osd memory target = 1610612736 2024-05-15T12:41:22.017 INFO:teuthology.orchestra.run.smithi117.stdout: osd objectstore = bluestore 2024-05-15T12:41:22.017 INFO:teuthology.orchestra.run.smithi117.stdout: osd op queue = debug_random 2024-05-15T12:41:22.017 INFO:teuthology.orchestra.run.smithi117.stdout: osd op queue cut off = debug_random 2024-05-15T12:41:22.017 INFO:teuthology.orchestra.run.smithi117.stdout: osd shutdown pgref assert = True 2024-05-15T12:41:22.017 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:41:22.017 INFO:teuthology.orchestra.run.smithi117.stdout:[mgr] 2024-05-15T12:41:22.017 INFO:teuthology.orchestra.run.smithi117.stdout: debug ms = 1 2024-05-15T12:41:22.017 INFO:teuthology.orchestra.run.smithi117.stdout: debug mgr = 20 2024-05-15T12:41:22.017 INFO:teuthology.orchestra.run.smithi117.stdout: debug mon = 20 2024-05-15T12:41:22.018 INFO:teuthology.orchestra.run.smithi117.stdout: debug auth = 20 2024-05-15T12:41:22.018 INFO:teuthology.orchestra.run.smithi117.stdout: mon reweight min pgs per osd = 4 2024-05-15T12:41:22.018 INFO:teuthology.orchestra.run.smithi117.stdout: mon reweight min bytes per osd = 10 2024-05-15T12:41:22.018 INFO:teuthology.orchestra.run.smithi117.stdout: mgr/telemetry/nag = false 2024-05-15T12:41:22.018 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:41:22.018 INFO:teuthology.orchestra.run.smithi117.stdout:[mon] 2024-05-15T12:41:22.018 INFO:teuthology.orchestra.run.smithi117.stdout: debug ms = 1 2024-05-15T12:41:22.018 INFO:teuthology.orchestra.run.smithi117.stdout: debug mon = 20 2024-05-15T12:41:22.018 INFO:teuthology.orchestra.run.smithi117.stdout: debug paxos = 20 2024-05-15T12:41:22.018 INFO:teuthology.orchestra.run.smithi117.stdout: debug auth = 20 2024-05-15T12:41:22.018 INFO:teuthology.orchestra.run.smithi117.stdout: mon data avail warn = 5 2024-05-15T12:41:22.018 INFO:teuthology.orchestra.run.smithi117.stdout: mon mgr mkfs grace = 240 2024-05-15T12:41:22.018 INFO:teuthology.orchestra.run.smithi117.stdout: mon reweight min pgs per osd = 4 2024-05-15T12:41:22.018 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd reporter subtree level = osd 2024-05-15T12:41:22.019 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd prime pg temp = true 2024-05-15T12:41:22.019 INFO:teuthology.orchestra.run.smithi117.stdout: mon reweight min bytes per osd = 10 2024-05-15T12:41:22.019 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:41:22.019 INFO:teuthology.orchestra.run.smithi117.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-05-15T12:41:22.019 INFO:teuthology.orchestra.run.smithi117.stdout: auth mon ticket ttl = 660 # 11m 2024-05-15T12:41:22.019 INFO:teuthology.orchestra.run.smithi117.stdout: auth service ticket ttl = 240 # 4m 2024-05-15T12:41:22.019 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:41:22.019 INFO:teuthology.orchestra.run.smithi117.stdout: # don't complain about insecure global_id in the test suite 2024-05-15T12:41:22.019 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-05-15T12:41:22.019 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-05-15T12:41:22.019 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:41:22.019 INFO:teuthology.orchestra.run.smithi117.stdout: # 1m isn't quite enough 2024-05-15T12:41:22.019 INFO:teuthology.orchestra.run.smithi117.stdout: mon_down_mkfs_grace = 2m 2024-05-15T12:41:22.019 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:41:22.019 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_filestore_osds = false 2024-05-15T12:41:22.020 INFO:teuthology.orchestra.run.smithi117.stdout: mon min osdmap epochs = 50 2024-05-15T12:41:22.020 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd crush smoke test = False 2024-05-15T12:41:22.020 INFO:teuthology.orchestra.run.smithi117.stdout: mon osdmap full prune interval = 2 2024-05-15T12:41:22.020 INFO:teuthology.orchestra.run.smithi117.stdout: mon osdmap full prune min = 15 2024-05-15T12:41:22.020 INFO:teuthology.orchestra.run.smithi117.stdout: mon osdmap full prune txsize = 2 2024-05-15T12:41:22.020 INFO:teuthology.orchestra.run.smithi117.stdout: mon scrub interval = 300 2024-05-15T12:41:22.020 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on pool no app = False 2024-05-15T12:41:22.020 INFO:teuthology.orchestra.run.smithi117.stdout: paxos service trim min = 10 2024-05-15T12:41:22.020 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:41:22.020 INFO:teuthology.orchestra.run.smithi117.stdout:[client] 2024-05-15T12:41:22.020 INFO:teuthology.orchestra.run.smithi117.stdout: rgw cache enabled = true 2024-05-15T12:41:22.020 INFO:teuthology.orchestra.run.smithi117.stdout: rgw enable ops log = true 2024-05-15T12:41:22.020 INFO:teuthology.orchestra.run.smithi117.stdout: rgw enable usage log = true 2024-05-15T12:41:22.020 INFO:teuthology.orchestra.run.smithi117.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-05-15T12:41:22.020 INFO:teuthology.orchestra.run.smithi117.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-05-15T12:41:22.020 INFO:teuthology.orchestra.run.smithi117.stdout: debug monc = 20 2024-05-15T12:41:22.021 INFO:teuthology.orchestra.run.smithi117.stdout: debug ms = 1 2024-05-15T12:41:22.021 INFO:teuthology.orchestra.run.smithi117.stdout: debug objecter = 20 2024-05-15T12:41:22.021 INFO:teuthology.orchestra.run.smithi117.stdout: debug rados = 20 2024-05-15T12:41:22.021 INFO:teuthology.orchestra.run.smithi117.stdout:[mon.a] 2024-05-15T12:41:22.021 INFO:teuthology.orchestra.run.smithi117.stdout:[mon.c] 2024-05-15T12:41:22.021 INFO:teuthology.orchestra.run.smithi117.stdout:[mon.b] 2024-05-15T12:41:22.021 INFO:teuthology.orchestra.run.smithi178.stdout:[global] 2024-05-15T12:41:22.022 INFO:teuthology.orchestra.run.smithi178.stdout: chdir = "" 2024-05-15T12:41:22.022 INFO:teuthology.orchestra.run.smithi178.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-05-15T12:41:22.022 INFO:teuthology.orchestra.run.smithi178.stdout: auth supported = cephx 2024-05-15T12:41:22.022 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:41:22.022 INFO:teuthology.orchestra.run.smithi178.stdout: filestore xattr use omap = true 2024-05-15T12:41:22.022 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:41:22.022 INFO:teuthology.orchestra.run.smithi178.stdout: mon clock drift allowed = 1.000 2024-05-15T12:41:22.022 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:41:22.022 INFO:teuthology.orchestra.run.smithi178.stdout: osd crush chooseleaf type = 0 2024-05-15T12:41:22.022 INFO:teuthology.orchestra.run.smithi178.stdout: auth debug = true 2024-05-15T12:41:22.022 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:41:22.022 INFO:teuthology.orchestra.run.smithi178.stdout: ms die on old message = true 2024-05-15T12:41:22.022 INFO:teuthology.orchestra.run.smithi178.stdout: ms die on bug = true 2024-05-15T12:41:22.022 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:41:22.023 INFO:teuthology.orchestra.run.smithi178.stdout: mon max pg per osd = 10000 # >= luminous 2024-05-15T12:41:22.023 INFO:teuthology.orchestra.run.smithi178.stdout: mon pg warn max object skew = 0 2024-05-15T12:41:22.023 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:41:22.023 INFO:teuthology.orchestra.run.smithi178.stdout: # disable pg_autoscaler by default for new pools 2024-05-15T12:41:22.023 INFO:teuthology.orchestra.run.smithi178.stdout: osd_pool_default_pg_autoscale_mode = off 2024-05-15T12:41:22.023 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:41:22.023 INFO:teuthology.orchestra.run.smithi178.stdout: osd pool default size = 2 2024-05-15T12:41:22.023 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:41:22.023 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd allow primary affinity = true 2024-05-15T12:41:22.023 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd allow pg remap = true 2024-05-15T12:41:22.023 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on legacy crush tunables = false 2024-05-15T12:41:22.023 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on crush straw calc version zero = false 2024-05-15T12:41:22.023 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on no sortbitwise = false 2024-05-15T12:41:22.023 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on osd down out interval zero = false 2024-05-15T12:41:22.023 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on too few osds = false 2024-05-15T12:41:22.024 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-05-15T12:41:22.024 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_pool_no_redundancy = false 2024-05-15T12:41:22.024 INFO:teuthology.orchestra.run.smithi178.stdout: mon_allow_pool_size_one = true 2024-05-15T12:41:22.024 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:41:22.024 INFO:teuthology.orchestra.run.smithi178.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-05-15T12:41:22.024 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:41:22.024 INFO:teuthology.orchestra.run.smithi178.stdout: osd default data pool replay window = 5 2024-05-15T12:41:22.024 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:41:22.024 INFO:teuthology.orchestra.run.smithi178.stdout: mon allow pool delete = true 2024-05-15T12:41:22.024 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:41:22.024 INFO:teuthology.orchestra.run.smithi178.stdout: mon cluster log file level = debug 2024-05-15T12:41:22.024 INFO:teuthology.orchestra.run.smithi178.stdout: debug asserts on shutdown = true 2024-05-15T12:41:22.024 INFO:teuthology.orchestra.run.smithi178.stdout: mon health detail to clog = false 2024-05-15T12:41:22.024 INFO:teuthology.orchestra.run.smithi178.stdout: mon host = "172.21.15.117,[v2:172.21.15.117:3301,v1:172.21.15.117:6790],172.21.15.178" 2024-05-15T12:41:22.024 INFO:teuthology.orchestra.run.smithi178.stdout: mon client directed command retry = 5 2024-05-15T12:41:22.025 INFO:teuthology.orchestra.run.smithi178.stdout: mon election default strategy = 1 2024-05-15T12:41:22.025 INFO:teuthology.orchestra.run.smithi178.stdout: ms inject socket failures = 5000 2024-05-15T12:41:22.025 INFO:teuthology.orchestra.run.smithi178.stdout: ms type = async 2024-05-15T12:41:22.025 INFO:teuthology.orchestra.run.smithi178.stdout: osd heartbeat grace = 80 2024-05-15T12:41:22.025 INFO:teuthology.orchestra.run.smithi178.stdout: fsid = 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:41:22.025 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:41:22.025 INFO:teuthology.orchestra.run.smithi178.stdout:[osd] 2024-05-15T12:41:22.025 INFO:teuthology.orchestra.run.smithi178.stdout: osd journal size = 100 2024-05-15T12:41:22.025 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:41:22.025 INFO:teuthology.orchestra.run.smithi178.stdout: osd scrub load threshold = 5.0 2024-05-15T12:41:22.025 INFO:teuthology.orchestra.run.smithi178.stdout: osd scrub max interval = 600 2024-05-15T12:41:22.025 INFO:teuthology.orchestra.run.smithi178.stdout: osd mclock profile = high_recovery_ops 2024-05-15T12:41:22.025 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:41:22.025 INFO:teuthology.orchestra.run.smithi178.stdout: osd recover clone overlap = true 2024-05-15T12:41:22.025 INFO:teuthology.orchestra.run.smithi178.stdout: osd recovery max chunk = 1048576 2024-05-15T12:41:22.026 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:41:22.026 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug shutdown = true 2024-05-15T12:41:22.026 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug op order = true 2024-05-15T12:41:22.026 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug verify stray on activate = true 2024-05-15T12:41:22.026 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug trim objects = true 2024-05-15T12:41:22.026 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:41:22.026 INFO:teuthology.orchestra.run.smithi178.stdout: osd open classes on start = true 2024-05-15T12:41:22.026 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug pg log writeout = true 2024-05-15T12:41:22.026 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:41:22.027 INFO:teuthology.orchestra.run.smithi178.stdout: osd deep scrub update digest min age = 30 2024-05-15T12:41:22.027 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:41:22.027 INFO:teuthology.orchestra.run.smithi178.stdout: osd map max advance = 10 2024-05-15T12:41:22.027 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:41:22.027 INFO:teuthology.orchestra.run.smithi178.stdout: journal zero on create = true 2024-05-15T12:41:22.027 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:41:22.027 INFO:teuthology.orchestra.run.smithi178.stdout: filestore ondisk finisher threads = 3 2024-05-15T12:41:22.027 INFO:teuthology.orchestra.run.smithi178.stdout: filestore apply finisher threads = 3 2024-05-15T12:41:22.027 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:41:22.027 INFO:teuthology.orchestra.run.smithi178.stdout: bdev debug aio = true 2024-05-15T12:41:22.027 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug misdirected ops = true 2024-05-15T12:41:22.027 INFO:teuthology.orchestra.run.smithi178.stdout: bdev async discard = True 2024-05-15T12:41:22.027 INFO:teuthology.orchestra.run.smithi178.stdout: bdev enable discard = True 2024-05-15T12:41:22.027 INFO:teuthology.orchestra.run.smithi178.stdout: bluestore block size = 96636764160 2024-05-15T12:41:22.027 INFO:teuthology.orchestra.run.smithi178.stdout: bluestore fsck on mount = True 2024-05-15T12:41:22.027 INFO:teuthology.orchestra.run.smithi178.stdout: bluestore zero block detection = True 2024-05-15T12:41:22.028 INFO:teuthology.orchestra.run.smithi178.stdout: debug bluefs = 1 2024-05-15T12:41:22.028 INFO:teuthology.orchestra.run.smithi178.stdout: debug bluestore = 1 2024-05-15T12:41:22.028 INFO:teuthology.orchestra.run.smithi178.stdout: debug monc = 20 2024-05-15T12:41:22.028 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2024-05-15T12:41:22.028 INFO:teuthology.orchestra.run.smithi178.stdout: debug osd = 20 2024-05-15T12:41:22.028 INFO:teuthology.orchestra.run.smithi178.stdout: debug rocksdb = 10 2024-05-15T12:41:22.028 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd backfillfull_ratio = 0.85 2024-05-15T12:41:22.028 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd full ratio = 0.9 2024-05-15T12:41:22.028 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd nearfull ratio = 0.8 2024-05-15T12:41:22.028 INFO:teuthology.orchestra.run.smithi178.stdout: osd class default list = * 2024-05-15T12:41:22.028 INFO:teuthology.orchestra.run.smithi178.stdout: osd class load list = * 2024-05-15T12:41:22.028 INFO:teuthology.orchestra.run.smithi178.stdout: osd client watch timeout = 120 2024-05-15T12:41:22.028 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug verify cached snaps = True 2024-05-15T12:41:22.028 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug verify missing on start = True 2024-05-15T12:41:22.028 INFO:teuthology.orchestra.run.smithi178.stdout: osd failsafe full ratio = 0.95 2024-05-15T12:41:22.029 INFO:teuthology.orchestra.run.smithi178.stdout: osd fast shutdown = False 2024-05-15T12:41:22.029 INFO:teuthology.orchestra.run.smithi178.stdout: osd mclock override recovery settings = True 2024-05-15T12:41:22.029 INFO:teuthology.orchestra.run.smithi178.stdout: osd memory target = 1610612736 2024-05-15T12:41:22.029 INFO:teuthology.orchestra.run.smithi178.stdout: osd objectstore = bluestore 2024-05-15T12:41:22.029 INFO:teuthology.orchestra.run.smithi178.stdout: osd op queue = debug_random 2024-05-15T12:41:22.029 INFO:teuthology.orchestra.run.smithi178.stdout: osd op queue cut off = debug_random 2024-05-15T12:41:22.029 INFO:teuthology.orchestra.run.smithi178.stdout: osd shutdown pgref assert = True 2024-05-15T12:41:22.029 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:41:22.029 INFO:teuthology.orchestra.run.smithi178.stdout:[mgr] 2024-05-15T12:41:22.029 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2024-05-15T12:41:22.029 INFO:teuthology.orchestra.run.smithi178.stdout: debug mgr = 20 2024-05-15T12:41:22.029 INFO:teuthology.orchestra.run.smithi178.stdout: debug mon = 20 2024-05-15T12:41:22.029 INFO:teuthology.orchestra.run.smithi178.stdout: debug auth = 20 2024-05-15T12:41:22.029 INFO:teuthology.orchestra.run.smithi178.stdout: mon reweight min pgs per osd = 4 2024-05-15T12:41:22.029 INFO:teuthology.orchestra.run.smithi178.stdout: mon reweight min bytes per osd = 10 2024-05-15T12:41:22.029 INFO:teuthology.orchestra.run.smithi178.stdout: mgr/telemetry/nag = false 2024-05-15T12:41:22.030 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:41:22.030 INFO:teuthology.orchestra.run.smithi178.stdout:[mon] 2024-05-15T12:41:22.030 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2024-05-15T12:41:22.030 INFO:teuthology.orchestra.run.smithi178.stdout: debug mon = 20 2024-05-15T12:41:22.030 INFO:teuthology.orchestra.run.smithi178.stdout: debug paxos = 20 2024-05-15T12:41:22.030 INFO:teuthology.orchestra.run.smithi178.stdout: debug auth = 20 2024-05-15T12:41:22.030 INFO:teuthology.orchestra.run.smithi178.stdout: mon data avail warn = 5 2024-05-15T12:41:22.030 INFO:teuthology.orchestra.run.smithi178.stdout: mon mgr mkfs grace = 240 2024-05-15T12:41:22.030 INFO:teuthology.orchestra.run.smithi178.stdout: mon reweight min pgs per osd = 4 2024-05-15T12:41:22.030 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd reporter subtree level = osd 2024-05-15T12:41:22.030 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd prime pg temp = true 2024-05-15T12:41:22.030 INFO:teuthology.orchestra.run.smithi178.stdout: mon reweight min bytes per osd = 10 2024-05-15T12:41:22.030 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:41:22.030 INFO:teuthology.orchestra.run.smithi178.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-05-15T12:41:22.031 INFO:teuthology.orchestra.run.smithi178.stdout: auth mon ticket ttl = 660 # 11m 2024-05-15T12:41:22.031 INFO:teuthology.orchestra.run.smithi178.stdout: auth service ticket ttl = 240 # 4m 2024-05-15T12:41:22.031 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:41:22.031 INFO:teuthology.orchestra.run.smithi178.stdout: # don't complain about insecure global_id in the test suite 2024-05-15T12:41:22.031 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-05-15T12:41:22.031 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-05-15T12:41:22.031 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:41:22.031 INFO:teuthology.orchestra.run.smithi178.stdout: # 1m isn't quite enough 2024-05-15T12:41:22.031 INFO:teuthology.orchestra.run.smithi178.stdout: mon_down_mkfs_grace = 2m 2024-05-15T12:41:22.031 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:41:22.031 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_filestore_osds = false 2024-05-15T12:41:22.031 INFO:teuthology.orchestra.run.smithi178.stdout: mon min osdmap epochs = 50 2024-05-15T12:41:22.031 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd crush smoke test = False 2024-05-15T12:41:22.031 INFO:teuthology.orchestra.run.smithi178.stdout: mon osdmap full prune interval = 2 2024-05-15T12:41:22.031 INFO:teuthology.orchestra.run.smithi178.stdout: mon osdmap full prune min = 15 2024-05-15T12:41:22.031 INFO:teuthology.orchestra.run.smithi178.stdout: mon osdmap full prune txsize = 2 2024-05-15T12:41:22.032 INFO:teuthology.orchestra.run.smithi178.stdout: mon scrub interval = 300 2024-05-15T12:41:22.032 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on pool no app = False 2024-05-15T12:41:22.032 INFO:teuthology.orchestra.run.smithi178.stdout: paxos service trim min = 10 2024-05-15T12:41:22.032 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-05-15T12:41:22.032 INFO:teuthology.orchestra.run.smithi178.stdout:[client] 2024-05-15T12:41:22.032 INFO:teuthology.orchestra.run.smithi178.stdout: rgw cache enabled = true 2024-05-15T12:41:22.032 INFO:teuthology.orchestra.run.smithi178.stdout: rgw enable ops log = true 2024-05-15T12:41:22.032 INFO:teuthology.orchestra.run.smithi178.stdout: rgw enable usage log = true 2024-05-15T12:41:22.032 INFO:teuthology.orchestra.run.smithi178.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-05-15T12:41:22.032 INFO:teuthology.orchestra.run.smithi178.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-05-15T12:41:22.032 INFO:teuthology.orchestra.run.smithi178.stdout: debug monc = 20 2024-05-15T12:41:22.032 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2024-05-15T12:41:22.032 INFO:teuthology.orchestra.run.smithi178.stdout: debug objecter = 20 2024-05-15T12:41:22.032 INFO:teuthology.orchestra.run.smithi178.stdout: debug rados = 20 2024-05-15T12:41:22.032 INFO:teuthology.orchestra.run.smithi178.stdout:[mon.a] 2024-05-15T12:41:22.033 INFO:teuthology.orchestra.run.smithi178.stdout:[mon.c] 2024-05-15T12:41:22.033 INFO:teuthology.orchestra.run.smithi178.stdout:[mon.b] 2024-05-15T12:41:22.034 INFO:tasks.ceph:Creating admin key on mon.a... 2024-05-15T12:41:22.034 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-05-15T12:41:22.128 INFO:tasks.ceph:Copying monmap to all nodes... 2024-05-15T12:41:22.128 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-05-15T12:41:22.128 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-05-15T12:41:22.143 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-05-15T12:41:22.143 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-05-15T12:41:22.199 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi117.front.sepia.ceph.com 2024-05-15T12:41:22.199 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-05-15T12:41:22.199 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/ceph/ceph.keyring 2024-05-15T12:41:22.199 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-05-15T12:41:22.272 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-05-15T12:41:22.272 DEBUG:teuthology.orchestra.run.smithi117:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-05-15T12:41:22.328 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi178.front.sepia.ceph.com 2024-05-15T12:41:22.328 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-05-15T12:41:22.328 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/ceph/ceph.keyring 2024-05-15T12:41:22.328 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-05-15T12:41:22.364 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-05-15T12:41:22.364 DEBUG:teuthology.orchestra.run.smithi178:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-05-15T12:41:22.420 INFO:tasks.ceph:Setting up mon nodes... 2024-05-15T12:41:22.421 INFO:tasks.ceph:Setting up mgr nodes... 2024-05-15T12:41:22.421 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-05-15T12:41:22.470 INFO:teuthology.orchestra.run.smithi117.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-05-15T12:41:22.473 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-05-15T12:41:22.535 INFO:teuthology.orchestra.run.smithi178.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-05-15T12:41:22.538 INFO:tasks.ceph:Setting up mds nodes... 2024-05-15T12:41:22.538 INFO:tasks.ceph_client:Setting up client nodes... 2024-05-15T12:41:22.538 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-05-15T12:41:22.578 INFO:teuthology.orchestra.run.smithi117.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-05-15T12:41:22.591 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-05-15T12:41:22.631 INFO:teuthology.orchestra.run.smithi178.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-05-15T12:41:22.643 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-05-15T12:41:22.644 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi117.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi178.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-05-15T12:41:22.644 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-05-15T12:41:22.668 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-15T12:41:22.668 INFO:tasks.ceph:role: osd.0 2024-05-15T12:41:22.668 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi117.front.sepia.ceph.com 2024-05-15T12:41:22.668 DEBUG:teuthology.orchestra.run.smithi117:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-05-15T12:41:22.740 INFO:teuthology.orchestra.run.smithi117.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-05-15T12:41:22.740 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-15T12:41:22.740 INFO:teuthology.orchestra.run.smithi117.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-15T12:41:22.740 INFO:teuthology.orchestra.run.smithi117.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-15T12:41:22.740 INFO:teuthology.orchestra.run.smithi117.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-15T12:41:22.740 INFO:teuthology.orchestra.run.smithi117.stdout: = sunit=0 swidth=0 blks 2024-05-15T12:41:22.740 INFO:teuthology.orchestra.run.smithi117.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-15T12:41:22.740 INFO:teuthology.orchestra.run.smithi117.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-15T12:41:22.741 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-15T12:41:22.741 INFO:teuthology.orchestra.run.smithi117.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-15T12:41:23.242 INFO:teuthology.orchestra.run.smithi117.stdout:Discarding blocks...Done. 2024-05-15T12:41:23.244 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi117.front.sepia.ceph.com -o noatime 2024-05-15T12:41:23.244 DEBUG:teuthology.orchestra.run.smithi117:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-05-15T12:41:23.377 DEBUG:teuthology.orchestra.run.smithi117:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-05-15T12:41:23.408 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-05-15T12:41:23.474 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-15T12:41:23.474 INFO:tasks.ceph:role: osd.1 2024-05-15T12:41:23.474 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi117.front.sepia.ceph.com 2024-05-15T12:41:23.474 DEBUG:teuthology.orchestra.run.smithi117:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-05-15T12:41:23.547 INFO:teuthology.orchestra.run.smithi117.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-05-15T12:41:23.547 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-15T12:41:23.547 INFO:teuthology.orchestra.run.smithi117.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-15T12:41:23.548 INFO:teuthology.orchestra.run.smithi117.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-15T12:41:23.548 INFO:teuthology.orchestra.run.smithi117.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-15T12:41:23.548 INFO:teuthology.orchestra.run.smithi117.stdout: = sunit=0 swidth=0 blks 2024-05-15T12:41:23.548 INFO:teuthology.orchestra.run.smithi117.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-15T12:41:23.548 INFO:teuthology.orchestra.run.smithi117.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-15T12:41:23.548 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-15T12:41:23.548 INFO:teuthology.orchestra.run.smithi117.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-15T12:41:24.009 INFO:teuthology.orchestra.run.smithi117.stdout:Discarding blocks...Done. 2024-05-15T12:41:24.011 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi117.front.sepia.ceph.com -o noatime 2024-05-15T12:41:24.011 DEBUG:teuthology.orchestra.run.smithi117:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-05-15T12:41:24.101 DEBUG:teuthology.orchestra.run.smithi117:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-05-15T12:41:24.131 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-05-15T12:41:24.196 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-15T12:41:24.196 INFO:tasks.ceph:role: osd.2 2024-05-15T12:41:24.196 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi117.front.sepia.ceph.com 2024-05-15T12:41:24.196 DEBUG:teuthology.orchestra.run.smithi117:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-05-15T12:41:24.269 INFO:teuthology.orchestra.run.smithi117.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-05-15T12:41:24.269 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-15T12:41:24.269 INFO:teuthology.orchestra.run.smithi117.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-15T12:41:24.269 INFO:teuthology.orchestra.run.smithi117.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-15T12:41:24.269 INFO:teuthology.orchestra.run.smithi117.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-15T12:41:24.269 INFO:teuthology.orchestra.run.smithi117.stdout: = sunit=0 swidth=0 blks 2024-05-15T12:41:24.269 INFO:teuthology.orchestra.run.smithi117.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-15T12:41:24.269 INFO:teuthology.orchestra.run.smithi117.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-15T12:41:24.269 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-15T12:41:24.270 INFO:teuthology.orchestra.run.smithi117.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-15T12:41:24.758 INFO:teuthology.orchestra.run.smithi117.stdout:Discarding blocks...Done. 2024-05-15T12:41:24.760 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi117.front.sepia.ceph.com -o noatime 2024-05-15T12:41:24.760 DEBUG:teuthology.orchestra.run.smithi117:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-05-15T12:41:24.891 DEBUG:teuthology.orchestra.run.smithi117:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-05-15T12:41:24.921 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-05-15T12:41:24.985 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-15T12:41:24.985 INFO:tasks.ceph:role: osd.3 2024-05-15T12:41:24.985 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi117.front.sepia.ceph.com 2024-05-15T12:41:24.986 DEBUG:teuthology.orchestra.run.smithi117:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-05-15T12:41:25.057 INFO:teuthology.orchestra.run.smithi117.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-05-15T12:41:25.057 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-15T12:41:25.057 INFO:teuthology.orchestra.run.smithi117.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-15T12:41:25.057 INFO:teuthology.orchestra.run.smithi117.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-15T12:41:25.057 INFO:teuthology.orchestra.run.smithi117.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-15T12:41:25.057 INFO:teuthology.orchestra.run.smithi117.stdout: = sunit=0 swidth=0 blks 2024-05-15T12:41:25.057 INFO:teuthology.orchestra.run.smithi117.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-15T12:41:25.058 INFO:teuthology.orchestra.run.smithi117.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-15T12:41:25.058 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-15T12:41:25.058 INFO:teuthology.orchestra.run.smithi117.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-15T12:41:25.589 INFO:teuthology.orchestra.run.smithi117.stdout:Discarding blocks...Done. 2024-05-15T12:41:25.591 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi117.front.sepia.ceph.com -o noatime 2024-05-15T12:41:25.591 DEBUG:teuthology.orchestra.run.smithi117:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-05-15T12:41:25.730 DEBUG:teuthology.orchestra.run.smithi117:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-05-15T12:41:25.762 DEBUG:teuthology.orchestra.run.smithi117:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-15T12:41:25.860 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:25.869+0000 7f77fba23640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-05-15T12:41:25.860 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:25.869+0000 7f77fba23640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-05-15T12:41:25.860 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:25.869+0000 7f77fba23640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-05-15T12:41:25.860 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:25.869+0000 7f77fba23640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-05-15T12:41:25.860 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:25.869+0000 7f77fba23640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-05-15T12:41:25.860 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:25.870+0000 7f77fba23640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-05-15T12:41:28.707 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-05-15T12:41:28.736 DEBUG:teuthology.orchestra.run.smithi117:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-15T12:41:28.826 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:28.835+0000 7fd33600d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-05-15T12:41:28.826 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:28.836+0000 7fd33600d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-05-15T12:41:28.826 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:28.836+0000 7fd33600d640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-05-15T12:41:28.826 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:28.836+0000 7fd33600d640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-05-15T12:41:28.826 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:28.836+0000 7fd33600d640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-05-15T12:41:28.826 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:28.836+0000 7fd33600d640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-05-15T12:41:31.669 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-05-15T12:41:31.699 DEBUG:teuthology.orchestra.run.smithi117:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-15T12:41:31.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:31.799+0000 7f807de12640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-05-15T12:41:31.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:31.800+0000 7f807de12640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-05-15T12:41:31.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:31.800+0000 7f807de12640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-05-15T12:41:31.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:31.800+0000 7f807de12640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-05-15T12:41:31.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:31.800+0000 7f807de12640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-05-15T12:41:31.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:31.801+0000 7f807de12640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-05-15T12:41:34.641 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-05-15T12:41:34.671 DEBUG:teuthology.orchestra.run.smithi117:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-15T12:41:34.764 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:34.774+0000 7f0b11015640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-05-15T12:41:34.764 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:34.774+0000 7f0b11015640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-05-15T12:41:34.764 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:34.774+0000 7f0b11015640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-05-15T12:41:34.764 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:34.774+0000 7f0b11015640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-05-15T12:41:34.765 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:34.774+0000 7f0b11015640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-05-15T12:41:34.765 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:34.774+0000 7f0b11015640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-05-15T12:41:37.607 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-05-15T12:41:37.636 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-05-15T12:41:37.666 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-15T12:41:37.666 INFO:tasks.ceph:role: osd.4 2024-05-15T12:41:37.666 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi178.front.sepia.ceph.com 2024-05-15T12:41:37.666 DEBUG:teuthology.orchestra.run.smithi178:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-05-15T12:41:37.735 INFO:teuthology.orchestra.run.smithi178.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-05-15T12:41:37.735 INFO:teuthology.orchestra.run.smithi178.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-15T12:41:37.735 INFO:teuthology.orchestra.run.smithi178.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-15T12:41:37.735 INFO:teuthology.orchestra.run.smithi178.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-15T12:41:37.735 INFO:teuthology.orchestra.run.smithi178.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-15T12:41:37.735 INFO:teuthology.orchestra.run.smithi178.stdout: = sunit=0 swidth=0 blks 2024-05-15T12:41:37.735 INFO:teuthology.orchestra.run.smithi178.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-15T12:41:37.735 INFO:teuthology.orchestra.run.smithi178.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-15T12:41:37.735 INFO:teuthology.orchestra.run.smithi178.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-15T12:41:37.735 INFO:teuthology.orchestra.run.smithi178.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-15T12:41:37.768 INFO:teuthology.orchestra.run.smithi178.stdout:Discarding blocks...Done. 2024-05-15T12:41:37.769 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi178.front.sepia.ceph.com -o noatime 2024-05-15T12:41:37.770 DEBUG:teuthology.orchestra.run.smithi178:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-05-15T12:41:37.842 DEBUG:teuthology.orchestra.run.smithi178:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-05-15T12:41:37.912 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-05-15T12:41:37.979 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-15T12:41:37.979 INFO:tasks.ceph:role: osd.5 2024-05-15T12:41:37.979 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi178.front.sepia.ceph.com 2024-05-15T12:41:37.979 DEBUG:teuthology.orchestra.run.smithi178:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-05-15T12:41:38.043 INFO:teuthology.orchestra.run.smithi178.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-05-15T12:41:38.043 INFO:teuthology.orchestra.run.smithi178.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-15T12:41:38.043 INFO:teuthology.orchestra.run.smithi178.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-15T12:41:38.043 INFO:teuthology.orchestra.run.smithi178.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-15T12:41:38.043 INFO:teuthology.orchestra.run.smithi178.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-15T12:41:38.043 INFO:teuthology.orchestra.run.smithi178.stdout: = sunit=0 swidth=0 blks 2024-05-15T12:41:38.044 INFO:teuthology.orchestra.run.smithi178.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-15T12:41:38.044 INFO:teuthology.orchestra.run.smithi178.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-15T12:41:38.044 INFO:teuthology.orchestra.run.smithi178.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-15T12:41:38.044 INFO:teuthology.orchestra.run.smithi178.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-15T12:41:38.075 INFO:teuthology.orchestra.run.smithi178.stdout:Discarding blocks...Done. 2024-05-15T12:41:38.077 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi178.front.sepia.ceph.com -o noatime 2024-05-15T12:41:38.077 DEBUG:teuthology.orchestra.run.smithi178:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-05-15T12:41:38.145 DEBUG:teuthology.orchestra.run.smithi178:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-05-15T12:41:38.217 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-05-15T12:41:38.281 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-15T12:41:38.282 INFO:tasks.ceph:role: osd.6 2024-05-15T12:41:38.282 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi178.front.sepia.ceph.com 2024-05-15T12:41:38.282 DEBUG:teuthology.orchestra.run.smithi178:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-05-15T12:41:38.346 INFO:teuthology.orchestra.run.smithi178.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-05-15T12:41:38.346 INFO:teuthology.orchestra.run.smithi178.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-15T12:41:38.346 INFO:teuthology.orchestra.run.smithi178.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-15T12:41:38.347 INFO:teuthology.orchestra.run.smithi178.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-15T12:41:38.347 INFO:teuthology.orchestra.run.smithi178.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-15T12:41:38.347 INFO:teuthology.orchestra.run.smithi178.stdout: = sunit=0 swidth=0 blks 2024-05-15T12:41:38.347 INFO:teuthology.orchestra.run.smithi178.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-15T12:41:38.347 INFO:teuthology.orchestra.run.smithi178.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-15T12:41:38.347 INFO:teuthology.orchestra.run.smithi178.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-15T12:41:38.347 INFO:teuthology.orchestra.run.smithi178.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-15T12:41:38.379 INFO:teuthology.orchestra.run.smithi178.stdout:Discarding blocks...Done. 2024-05-15T12:41:38.381 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi178.front.sepia.ceph.com -o noatime 2024-05-15T12:41:38.381 DEBUG:teuthology.orchestra.run.smithi178:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-05-15T12:41:38.450 DEBUG:teuthology.orchestra.run.smithi178:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-05-15T12:41:38.521 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-05-15T12:41:38.586 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-15T12:41:38.586 INFO:tasks.ceph:role: osd.7 2024-05-15T12:41:38.586 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi178.front.sepia.ceph.com 2024-05-15T12:41:38.586 DEBUG:teuthology.orchestra.run.smithi178:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-05-15T12:41:38.650 INFO:teuthology.orchestra.run.smithi178.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-05-15T12:41:38.650 INFO:teuthology.orchestra.run.smithi178.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-15T12:41:38.650 INFO:teuthology.orchestra.run.smithi178.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-15T12:41:38.650 INFO:teuthology.orchestra.run.smithi178.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-15T12:41:38.650 INFO:teuthology.orchestra.run.smithi178.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-15T12:41:38.650 INFO:teuthology.orchestra.run.smithi178.stdout: = sunit=0 swidth=0 blks 2024-05-15T12:41:38.650 INFO:teuthology.orchestra.run.smithi178.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-15T12:41:38.650 INFO:teuthology.orchestra.run.smithi178.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-15T12:41:38.650 INFO:teuthology.orchestra.run.smithi178.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-15T12:41:38.650 INFO:teuthology.orchestra.run.smithi178.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-15T12:41:38.682 INFO:teuthology.orchestra.run.smithi178.stdout:Discarding blocks...Done. 2024-05-15T12:41:38.684 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi178.front.sepia.ceph.com -o noatime 2024-05-15T12:41:38.684 DEBUG:teuthology.orchestra.run.smithi178:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-05-15T12:41:38.753 DEBUG:teuthology.orchestra.run.smithi178:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-05-15T12:41:38.825 DEBUG:teuthology.orchestra.run.smithi178:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-15T12:41:38.950 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:41:38.959+0000 7f756ee20640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-05-15T12:41:38.950 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:41:38.960+0000 7f756ee20640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-05-15T12:41:38.950 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:41:38.960+0000 7f756ee20640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-05-15T12:41:38.950 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:41:38.960+0000 7f756ee20640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-05-15T12:41:38.950 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:41:38.960+0000 7f756ee20640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-05-15T12:41:38.950 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:41:38.960+0000 7f756ee20640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-05-15T12:41:41.797 DEBUG:teuthology.orchestra.run.smithi178:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-05-15T12:41:41.826 DEBUG:teuthology.orchestra.run.smithi178:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-15T12:41:41.916 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:41:41.926+0000 7f0f23012640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-05-15T12:41:41.916 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:41:41.926+0000 7f0f23012640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-05-15T12:41:41.916 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:41:41.926+0000 7f0f23012640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-05-15T12:41:41.916 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:41:41.926+0000 7f0f23012640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-05-15T12:41:41.916 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:41:41.926+0000 7f0f23012640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-05-15T12:41:41.916 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:41:41.926+0000 7f0f23012640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-05-15T12:41:44.775 DEBUG:teuthology.orchestra.run.smithi178:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-05-15T12:41:44.803 DEBUG:teuthology.orchestra.run.smithi178:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-15T12:41:44.894 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:41:44.903+0000 7f0892c1d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-05-15T12:41:44.894 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:41:44.904+0000 7f0892c1d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-05-15T12:41:44.894 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:41:44.904+0000 7f0892c1d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-05-15T12:41:44.894 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:41:44.904+0000 7f0892c1d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-05-15T12:41:44.894 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:41:44.904+0000 7f0892c1d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-05-15T12:41:44.894 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:41:44.904+0000 7f0892c1d640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-05-15T12:41:47.736 DEBUG:teuthology.orchestra.run.smithi178:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-05-15T12:41:47.765 DEBUG:teuthology.orchestra.run.smithi178:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-15T12:41:47.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:41:47.865+0000 7f0c42f61640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-05-15T12:41:47.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:41:47.865+0000 7f0c42f61640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-05-15T12:41:47.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:41:47.865+0000 7f0c42f61640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-05-15T12:41:47.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:41:47.865+0000 7f0c42f61640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-05-15T12:41:47.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:41:47.865+0000 7f0c42f61640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-05-15T12:41:47.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:41:47.865+0000 7f0c42f61640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-05-15T12:41:50.705 DEBUG:teuthology.orchestra.run.smithi178:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-05-15T12:41:50.734 INFO:tasks.ceph:Reading keys from all nodes... 2024-05-15T12:41:50.734 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-05-15T12:41:50.735 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-05-15T12:41:50.765 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-05-15T12:41:50.765 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-05-15T12:41:50.834 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-05-15T12:41:50.835 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-05-15T12:41:50.905 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-05-15T12:41:50.905 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-05-15T12:41:50.973 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-05-15T12:41:50.974 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-05-15T12:41:51.042 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-05-15T12:41:51.043 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-05-15T12:41:51.072 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-05-15T12:41:51.073 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-05-15T12:41:51.142 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-05-15T12:41:51.142 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-05-15T12:41:51.211 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-05-15T12:41:51.211 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-05-15T12:41:51.278 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-05-15T12:41:51.278 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-05-15T12:41:51.345 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-05-15T12:41:51.345 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-05-15T12:41:51.364 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-05-15T12:41:51.364 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-05-15T12:41:51.401 INFO:tasks.ceph:Adding keys to all mons... 2024-05-15T12:41:51.402 DEBUG:teuthology.orchestra.run.smithi117:> sudo tee -a /etc/ceph/ceph.keyring 2024-05-15T12:41:51.407 DEBUG:teuthology.orchestra.run.smithi178:> sudo tee -a /etc/ceph/ceph.keyring 2024-05-15T12:41:51.445 INFO:teuthology.orchestra.run.smithi117.stdout:[mgr.y] 2024-05-15T12:41:51.445 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQByrURm3sC8HBAAxbsCGkc3nGZ9EXLejYZasA== 2024-05-15T12:41:51.446 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.0] 2024-05-15T12:41:51.446 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQB1rURmcBLkMxAAyfZ4ougsFXjbIiAUQl/nJw== 2024-05-15T12:41:51.446 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.1] 2024-05-15T12:41:51.446 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQB4rURmn/zgMRAATsvnkRl5+7rONz91e4fw2w== 2024-05-15T12:41:51.446 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.2] 2024-05-15T12:41:51.446 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQB7rURmRYu+LxAApD4ezgB6qYXQ/oj6vSlxvg== 2024-05-15T12:41:51.446 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.3] 2024-05-15T12:41:51.446 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQB+rURmQWozLhAAe0wNy9vr1IAhZgpRK77z8Q== 2024-05-15T12:41:51.446 INFO:teuthology.orchestra.run.smithi117.stdout:[mgr.x] 2024-05-15T12:41:51.446 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQByrURmrfuZIBAAEnSLxMvMmt6RI33eR1jNGQ== 2024-05-15T12:41:51.446 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.4] 2024-05-15T12:41:51.446 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQCCrURmvbpFORAAmARXIuJ4cEit9cpc0mAWuw== 2024-05-15T12:41:51.446 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.5] 2024-05-15T12:41:51.447 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQCFrURm0WtBNxAA4dOGKHAxUPiMkmlLQYumUw== 2024-05-15T12:41:51.447 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.6] 2024-05-15T12:41:51.447 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQCIrURmY+HuNRAAPu7bFX1zZeuxBzInbkrqPA== 2024-05-15T12:41:51.447 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.7] 2024-05-15T12:41:51.447 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQCLrURmIW2hMxAAqyeVcm1YL3lj0L0kE1LzNw== 2024-05-15T12:41:51.447 INFO:teuthology.orchestra.run.smithi117.stdout:[client.0] 2024-05-15T12:41:51.447 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQByrURmTJ0uIxAAG85X82iXRYLNxauaC6WG/A== 2024-05-15T12:41:51.447 INFO:teuthology.orchestra.run.smithi117.stdout:[client.1] 2024-05-15T12:41:51.447 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQByrURmPutPJhAAgQenfa/6O5dKiMDP/vA58A== 2024-05-15T12:41:51.465 INFO:teuthology.orchestra.run.smithi178.stdout:[mgr.y] 2024-05-15T12:41:51.465 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQByrURm3sC8HBAAxbsCGkc3nGZ9EXLejYZasA== 2024-05-15T12:41:51.465 INFO:teuthology.orchestra.run.smithi178.stdout:[osd.0] 2024-05-15T12:41:51.465 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQB1rURmcBLkMxAAyfZ4ougsFXjbIiAUQl/nJw== 2024-05-15T12:41:51.465 INFO:teuthology.orchestra.run.smithi178.stdout:[osd.1] 2024-05-15T12:41:51.465 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQB4rURmn/zgMRAATsvnkRl5+7rONz91e4fw2w== 2024-05-15T12:41:51.465 INFO:teuthology.orchestra.run.smithi178.stdout:[osd.2] 2024-05-15T12:41:51.465 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQB7rURmRYu+LxAApD4ezgB6qYXQ/oj6vSlxvg== 2024-05-15T12:41:51.465 INFO:teuthology.orchestra.run.smithi178.stdout:[osd.3] 2024-05-15T12:41:51.466 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQB+rURmQWozLhAAe0wNy9vr1IAhZgpRK77z8Q== 2024-05-15T12:41:51.466 INFO:teuthology.orchestra.run.smithi178.stdout:[mgr.x] 2024-05-15T12:41:51.466 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQByrURmrfuZIBAAEnSLxMvMmt6RI33eR1jNGQ== 2024-05-15T12:41:51.466 INFO:teuthology.orchestra.run.smithi178.stdout:[osd.4] 2024-05-15T12:41:51.466 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQCCrURmvbpFORAAmARXIuJ4cEit9cpc0mAWuw== 2024-05-15T12:41:51.466 INFO:teuthology.orchestra.run.smithi178.stdout:[osd.5] 2024-05-15T12:41:51.466 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQCFrURm0WtBNxAA4dOGKHAxUPiMkmlLQYumUw== 2024-05-15T12:41:51.466 INFO:teuthology.orchestra.run.smithi178.stdout:[osd.6] 2024-05-15T12:41:51.466 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQCIrURmY+HuNRAAPu7bFX1zZeuxBzInbkrqPA== 2024-05-15T12:41:51.466 INFO:teuthology.orchestra.run.smithi178.stdout:[osd.7] 2024-05-15T12:41:51.466 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQCLrURmIW2hMxAAqyeVcm1YL3lj0L0kE1LzNw== 2024-05-15T12:41:51.466 INFO:teuthology.orchestra.run.smithi178.stdout:[client.0] 2024-05-15T12:41:51.466 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQByrURmTJ0uIxAAG85X82iXRYLNxauaC6WG/A== 2024-05-15T12:41:51.466 INFO:teuthology.orchestra.run.smithi178.stdout:[client.1] 2024-05-15T12:41:51.466 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQByrURmPutPJhAAgQenfa/6O5dKiMDP/vA58A== 2024-05-15T12:41:51.467 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-15T12:41:51.490 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-15T12:41:51.558 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-15T12:41:51.560 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-15T12:41:51.611 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-15T12:41:51.614 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-15T12:41:51.665 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-15T12:41:51.668 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-15T12:41:51.719 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-15T12:41:51.721 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-15T12:41:51.773 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-15T12:41:51.778 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-15T12:41:51.829 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-15T12:41:51.831 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-15T12:41:51.881 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-15T12:41:51.884 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-15T12:41:51.934 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-15T12:41:51.937 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-15T12:41:51.987 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-15T12:41:51.989 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-15T12:41:52.040 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-15T12:41:52.042 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-15T12:41:52.096 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-15T12:41:52.098 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-15T12:41:52.151 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-05-15T12:41:52.151 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-05-15T12:41:52.176 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-15T12:41:52.272 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-05-15T12:41:52.296 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-05-15T12:41:52.361 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-15T12:41:52.457 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-05-15T12:41:52.481 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-05-15T12:41:52.506 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-15T12:41:52.601 DEBUG:teuthology.orchestra.run.smithi178:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-05-15T12:41:52.626 DEBUG:teuthology.orchestra.run.smithi117:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-05-15T12:41:52.628 DEBUG:teuthology.orchestra.run.smithi178:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-05-15T12:41:52.681 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-05-15T12:41:52.682 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-05-15T12:41:52.682 INFO:tasks.ceph.mon.a:Restarting daemon 2024-05-15T12:41:52.682 DEBUG:teuthology.orchestra.run.smithi117:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-05-15T12:41:52.684 INFO:tasks.ceph.mon.a:Started 2024-05-15T12:41:52.685 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-05-15T12:41:52.685 INFO:tasks.ceph.mon.c:Restarting daemon 2024-05-15T12:41:52.685 DEBUG:teuthology.orchestra.run.smithi117:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-05-15T12:41:52.687 INFO:tasks.ceph.mon.c:Started 2024-05-15T12:41:52.687 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-05-15T12:41:52.687 INFO:tasks.ceph.mon.b:Restarting daemon 2024-05-15T12:41:52.687 DEBUG:teuthology.orchestra.run.smithi178:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-05-15T12:41:52.724 INFO:tasks.ceph.mon.b:Started 2024-05-15T12:41:52.724 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-05-15T12:41:52.724 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-05-15T12:41:52.724 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-05-15T12:41:52.727 INFO:tasks.ceph.mgr.y:Started 2024-05-15T12:41:52.727 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-05-15T12:41:52.727 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-05-15T12:41:52.729 INFO:tasks.ceph.mgr.x:Started 2024-05-15T12:41:52.729 INFO:tasks.ceph:Setting crush tunables to default 2024-05-15T12:41:52.729 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph osd crush tunables default 2024-05-15T12:41:52.828 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:52.838+0000 7fc94e4c8640 10 monclient: get_monmap_and_config 2024-05-15T12:41:52.828 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:52.838+0000 7fc94e4c8640 10 monclient: build_initial_monmap 2024-05-15T12:41:52.828 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:52.838+0000 7fc94e4c8640 10 monclient: monmap: 2024-05-15T12:41:52.828 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:41:52.828 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:41:52.829 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:52.839622+0000 2024-05-15T12:41:52.829 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:52.839622+0000 2024-05-15T12:41:52.829 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:41:52.829 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:41:52.829 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T12:41:52.829 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T12:41:52.829 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T12:41:52.829 INFO:teuthology.orchestra.run.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T12:41:52.829 INFO:teuthology.orchestra.run.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T12:41:52.829 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:41:52.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:52.839+0000 7fc94e4c8640 1 Processor -- start 2024-05-15T12:41:52.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:52.839+0000 7fc94e4c8640 1 -- start start 2024-05-15T12:41:52.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:52.839+0000 7fc94e4c8640 10 monclient: init 2024-05-15T12:41:52.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:52.839+0000 7fc94e4c8640 10 monclient: _reopen_session rank -1 2024-05-15T12:41:52.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:52.839+0000 7fc94e4c8640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-05-15T12:41:52.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:52.839+0000 7fc94e4c8640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc9480b4d00 0x7fc9480b1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:41:52.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:52.839+0000 7fc94e4c8640 10 monclient(hunting): picked mon.noname-b con 0x7fc9480b4d00 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:41:52.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:52.839+0000 7fc94e4c8640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fc9480b50e0 0x7fc948145c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:41:52.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:52.839+0000 7fc94e4c8640 10 monclient(hunting): picked mon.noname-c con 0x7fc9480b50e0 addr v2:172.21.15.178:3300/0 2024-05-15T12:41:52.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:52.840+0000 7fc94e4c8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc9480a94b0 addr v1:172.21.15.117:6789/0 2024-05-15T12:41:52.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:52.840+0000 7fc94e4c8640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9480ab1b0 con 0x7fc9480a94b0 2024-05-15T12:41:52.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:52.840+0000 7fc94e4c8640 10 monclient(hunting): start opening mon connection 2024-05-15T12:41:52.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:52.840+0000 7fc94e4c8640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9480b23c0 con 0x7fc9480b50e0 2024-05-15T12:41:52.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:52.840+0000 7fc94e4c8640 10 monclient(hunting): start opening mon connection 2024-05-15T12:41:52.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:52.840+0000 7fc94e4c8640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9480b16f0 con 0x7fc9480b4d00 2024-05-15T12:41:52.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:52.840+0000 7fc94e4c8640 10 monclient(hunting): _renew_subs 2024-05-15T12:41:52.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:52.840+0000 7fc94e4c8640 10 monclient(hunting): authenticate will time out at 1276.081848s 2024-05-15T12:41:52.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:52.840+0000 7fc94ca65640 1 -- >> v1:172.21.15.117:6789/0 conn(0x7fc9480a94b0 legacy=0x7fc9480b0fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.117:6789/0 2024-05-15T12:41:52.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:52.840+0000 7fc947fff640 1 -- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc9480b4d00 msgr2=0x7fc9480b1e80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3301/0 2024-05-15T12:41:52.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:52.840+0000 7fc9477fe640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7fc9480b50e0 msgr2=0x7fc948145c20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2024-05-15T12:41:52.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:52.840+0000 7fc9477fe640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fc9480b50e0 0x7fc948145c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-15T12:41:52.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:52.840+0000 7fc947fff640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc9480b4d00 0x7fc9480b1e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-15T12:41:53.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:53.041+0000 7fc947fff640 1 -- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc9480b4d00 msgr2=0x7fc9480b1e80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3301/0 2024-05-15T12:41:53.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:53.041+0000 7fc94ca65640 1 -- >> v1:172.21.15.117:6789/0 conn(0x7fc9480a94b0 legacy=0x7fc9480b0fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.117:6789/0 2024-05-15T12:41:53.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:53.041+0000 7fc947fff640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc9480b4d00 0x7fc9480b1e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-15T12:41:53.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:53.041+0000 7fc9477fe640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7fc9480b50e0 msgr2=0x7fc948145c20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2024-05-15T12:41:53.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:53.041+0000 7fc9477fe640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fc9480b50e0 0x7fc948145c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-15T12:41:53.432 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:53.442+0000 7fc947fff640 1 -- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc9480b4d00 msgr2=0x7fc9480b1e80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3301/0 2024-05-15T12:41:53.432 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:53.442+0000 7fc94ca65640 1 -- >> v1:172.21.15.117:6789/0 conn(0x7fc9480a94b0 legacy=0x7fc9480b0fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.117:6789/0 2024-05-15T12:41:53.432 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:53.442+0000 7fc947fff640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc9480b4d00 0x7fc9480b1e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-15T12:41:53.432 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:53.442+0000 7fc9477fe640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7fc9480b50e0 msgr2=0x7fc948145c20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2024-05-15T12:41:53.432 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:53.442+0000 7fc9477fe640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fc9480b50e0 0x7fc948145c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-15T12:41:54.233 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:54.243+0000 7fc947fff640 1 -- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc9480b4d00 msgr2=0x7fc9480b1e80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3301/0 2024-05-15T12:41:54.233 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:54.243+0000 7fc94ca65640 1 -- >> v1:172.21.15.117:6789/0 conn(0x7fc9480a94b0 legacy=0x7fc9480b0fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.117:6789/0 2024-05-15T12:41:54.233 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:54.243+0000 7fc9477fe640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7fc9480b50e0 msgr2=0x7fc948145c20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2024-05-15T12:41:54.233 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:54.243+0000 7fc947fff640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc9480b4d00 0x7fc9480b1e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-15T12:41:54.233 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:54.243+0000 7fc9477fe640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fc9480b50e0 0x7fc948145c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-15T12:41:55.830 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:55.840+0000 7fc945ffb640 10 monclient(hunting): tick 2024-05-15T12:41:55.830 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:55.840+0000 7fc945ffb640 10 monclient(hunting): _check_auth_tickets 2024-05-15T12:41:55.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:55.840+0000 7fc945ffb640 1 monclient(hunting): continuing hunt 2024-05-15T12:41:55.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:55.840+0000 7fc945ffb640 10 monclient(hunting): _reopen_session rank -1 2024-05-15T12:41:55.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:55.840+0000 7fc945ffb640 1 -- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc9480b4d00 msgr2=0x7fc9480b1e80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-15T12:41:55.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:55.840+0000 7fc945ffb640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc9480b4d00 0x7fc9480b1e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:41:55.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:55.840+0000 7fc945ffb640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7fc9480b50e0 msgr2=0x7fc948145c20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-15T12:41:55.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:55.840+0000 7fc945ffb640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fc9480b50e0 0x7fc948145c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:41:55.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:55.840+0000 7fc945ffb640 1 -- >> v1:172.21.15.117:6789/0 conn(0x7fc9480a94b0 legacy=0x7fc9480b0fe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-15T12:41:55.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:55.840+0000 7fc945ffb640 10 monclient: _add_conns ranks=[1,3] 2024-05-15T12:41:55.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:55.840+0000 7fc945ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc9340036f0 addr v1:172.21.15.178:6789/0 2024-05-15T12:41:55.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:55.840+0000 7fc945ffb640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fc9340073a0 0x7fc934009860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:41:55.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:55.840+0000 7fc945ffb640 10 monclient(hunting): picked mon.noname-a con 0x7fc9340073a0 addr v2:172.21.15.117:3300/0 2024-05-15T12:41:55.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:55.840+0000 7fc945ffb640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc948146160 con 0x7fc9340036f0 2024-05-15T12:41:55.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:55.840+0000 7fc945ffb640 10 monclient(hunting): start opening mon connection 2024-05-15T12:41:55.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:55.840+0000 7fc945ffb640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9480b23c0 con 0x7fc9340073a0 2024-05-15T12:41:55.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:55.840+0000 7fc945ffb640 10 monclient(hunting): _renew_subs 2024-05-15T12:41:55.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:55.840+0000 7fc9477fe640 1 -- >> v2:172.21.15.117:3300/0 conn(0x7fc9340073a0 msgr2=0x7fc934009860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3300/0 2024-05-15T12:41:55.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:55.840+0000 7fc9477fe640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fc9340073a0 0x7fc934009860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-15T12:41:55.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:55.840+0000 7fc947fff640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7fc9340036f0 legacy=0x7fc934005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2024-05-15T12:41:56.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:56.041+0000 7fc9477fe640 1 -- >> v2:172.21.15.117:3300/0 conn(0x7fc9340073a0 msgr2=0x7fc934009860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3300/0 2024-05-15T12:41:56.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:56.041+0000 7fc9477fe640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fc9340073a0 0x7fc934009860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-15T12:41:56.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:56.041+0000 7fc947fff640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7fc9340036f0 legacy=0x7fc934005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2024-05-15T12:41:56.432 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:56.442+0000 7fc9477fe640 1 -- >> v2:172.21.15.117:3300/0 conn(0x7fc9340073a0 msgr2=0x7fc934009860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3300/0 2024-05-15T12:41:56.432 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:56.442+0000 7fc9477fe640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fc9340073a0 0x7fc934009860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-15T12:41:56.432 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:56.442+0000 7fc947fff640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7fc9340036f0 legacy=0x7fc934005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2024-05-15T12:41:57.233 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:57.243+0000 7fc9477fe640 1 -- >> v2:172.21.15.117:3300/0 conn(0x7fc9340073a0 msgr2=0x7fc934009860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3300/0 2024-05-15T12:41:57.233 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:57.243+0000 7fc9477fe640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fc9340073a0 0x7fc934009860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-15T12:41:57.233 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:57.243+0000 7fc947fff640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7fc9340036f0 legacy=0x7fc934005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2024-05-15T12:41:58.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:58.840+0000 7fc945ffb640 10 monclient(hunting): tick 2024-05-15T12:41:58.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:58.840+0000 7fc945ffb640 10 monclient(hunting): _check_auth_tickets 2024-05-15T12:41:58.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:58.840+0000 7fc945ffb640 1 monclient(hunting): continuing hunt 2024-05-15T12:41:58.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:58.840+0000 7fc945ffb640 10 monclient(hunting): _reopen_session rank -1 2024-05-15T12:41:58.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:58.840+0000 7fc945ffb640 1 -- >> v2:172.21.15.117:3300/0 conn(0x7fc9340073a0 msgr2=0x7fc934009860 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-15T12:41:58.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:58.840+0000 7fc945ffb640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fc9340073a0 0x7fc934009860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:41:58.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:58.840+0000 7fc945ffb640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7fc9340036f0 legacy=0x7fc934005b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-15T12:41:58.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:58.840+0000 7fc94ca65640 1 -- reap_dead start 2024-05-15T12:41:58.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:58.840+0000 7fc945ffb640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-05-15T12:41:58.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:58.840+0000 7fc945ffb640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fc934009f10 0x7fc934009860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:41:58.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:58.840+0000 7fc945ffb640 10 monclient(hunting): picked mon.noname-a con 0x7fc934009f10 addr v2:172.21.15.117:3300/0 2024-05-15T12:41:58.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:58.840+0000 7fc945ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc93400a3c0 addr v1:172.21.15.178:6789/0 2024-05-15T12:41:58.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:58.840+0000 7fc945ffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc93400a930 addr v1:172.21.15.117:6789/0 2024-05-15T12:41:58.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:58.840+0000 7fc945ffb640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9480ab1b0 con 0x7fc93400a930 2024-05-15T12:41:58.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:58.840+0000 7fc945ffb640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc948146160 con 0x7fc93400a3c0 2024-05-15T12:41:58.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:58.840+0000 7fc945ffb640 10 monclient(hunting): start opening mon connection 2024-05-15T12:41:58.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:58.840+0000 7fc945ffb640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9480b23c0 con 0x7fc934009f10 2024-05-15T12:41:58.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:58.840+0000 7fc945ffb640 10 monclient(hunting): _renew_subs 2024-05-15T12:41:58.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:58.840+0000 7fc94ca65640 1 -- >> v1:172.21.15.117:6789/0 conn(0x7fc93400a930 legacy=0x7fc93400d0c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.117:6789/0 2024-05-15T12:41:58.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:58.840+0000 7fc947fff640 1 -- >> v2:172.21.15.117:3300/0 conn(0x7fc934009f10 msgr2=0x7fc934009860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3300/0 2024-05-15T12:41:58.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:58.840+0000 7fc947fff640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fc934009f10 0x7fc934009860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-15T12:41:58.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:58.840+0000 7fc9477fe640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7fc93400a3c0 legacy=0x7fc934005ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2024-05-15T12:41:59.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:59.041+0000 7fc94ca65640 1 -- >> v1:172.21.15.117:6789/0 conn(0x7fc93400a930 legacy=0x7fc93400d0c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.117:6789/0 2024-05-15T12:41:59.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:59.041+0000 7fc947fff640 1 -- >> v2:172.21.15.117:3300/0 conn(0x7fc934009f10 msgr2=0x7fc934009860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3300/0 2024-05-15T12:41:59.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:59.041+0000 7fc947fff640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fc934009f10 0x7fc934009860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-15T12:41:59.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:59.041+0000 7fc9477fe640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7fc93400a3c0 legacy=0x7fc934005ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2024-05-15T12:41:59.432 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:59.442+0000 7fc94ca65640 1 -- >> v1:172.21.15.117:6789/0 conn(0x7fc93400a930 legacy=0x7fc93400d0c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.117:6789/0 2024-05-15T12:41:59.432 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:59.442+0000 7fc947fff640 1 -- >> v2:172.21.15.117:3300/0 conn(0x7fc934009f10 msgr2=0x7fc934009860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3300/0 2024-05-15T12:41:59.432 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:59.442+0000 7fc947fff640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fc934009f10 0x7fc934009860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-15T12:41:59.432 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:41:59.442+0000 7fc9477fe640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7fc93400a3c0 legacy=0x7fc934005ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2024-05-15T12:41:59.776 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:00:06.948 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:42:00.060 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:00:07.239 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:42:00.061 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:00:07.239 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:42:00.083 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:00:07.255 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:42:00.233 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:00.243+0000 7fc94ca65640 1 -- >> v1:172.21.15.117:6789/0 conn(0x7fc93400a930 legacy=0x7fc93400d0c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.117:6789/0 2024-05-15T12:42:00.233 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:00.243+0000 7fc947fff640 1 -- >> v2:172.21.15.117:3300/0 conn(0x7fc934009f10 msgr2=0x7fc934009860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3300/0 2024-05-15T12:42:00.233 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:00.243+0000 7fc9477fe640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7fc93400a3c0 legacy=0x7fc934005ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2024-05-15T12:42:00.233 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:00.243+0000 7fc947fff640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fc934009f10 0x7fc934009860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-15T12:42:00.381 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:00:07.559 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:42:00.381 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:00:07.559 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:42:01.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:01.840+0000 7fc945ffb640 10 monclient(hunting): tick 2024-05-15T12:42:01.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:01.840+0000 7fc945ffb640 10 monclient(hunting): _check_auth_tickets 2024-05-15T12:42:01.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:01.840+0000 7fc945ffb640 1 monclient(hunting): continuing hunt 2024-05-15T12:42:01.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:01.840+0000 7fc945ffb640 10 monclient(hunting): _reopen_session rank -1 2024-05-15T12:42:01.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:01.840+0000 7fc945ffb640 1 -- >> v2:172.21.15.117:3300/0 conn(0x7fc934009f10 msgr2=0x7fc934009860 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-15T12:42:01.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:01.840+0000 7fc945ffb640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fc934009f10 0x7fc934009860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:01.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:01.840+0000 7fc945ffb640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7fc93400a3c0 legacy=0x7fc934005ae0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-15T12:42:01.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:01.840+0000 7fc945ffb640 1 -- >> v1:172.21.15.117:6789/0 conn(0x7fc93400a930 legacy=0x7fc93400d0c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-15T12:42:01.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:01.840+0000 7fc945ffb640 10 monclient: _add_conns ranks=[4,2] 2024-05-15T12:42:01.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:01.840+0000 7fc945ffb640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fc93400f8e0 0x7fc93400fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:01.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:01.840+0000 7fc945ffb640 10 monclient(hunting): picked mon.noname-c con 0x7fc93400f8e0 addr v2:172.21.15.178:3300/0 2024-05-15T12:42:01.833 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:01.840+0000 7fc945ffb640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc934010360 0x7fc93400e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:01.833 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:01.840+0000 7fc945ffb640 10 monclient(hunting): picked mon.noname-b con 0x7fc934010360 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:42:01.833 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:01.840+0000 7fc945ffb640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:01.833 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:01.840+0000 7fc945ffb640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9480b23c0 con 0x7fc93400f8e0 2024-05-15T12:42:01.833 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:01.840+0000 7fc945ffb640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:01.833 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:01.840+0000 7fc945ffb640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9480b16f0 con 0x7fc934010360 2024-05-15T12:42:01.833 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:01.840+0000 7fc945ffb640 10 monclient(hunting): _renew_subs 2024-05-15T12:42:01.846 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:01.857+0000 7fc9477fe640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc934010360 0x7fc93400e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:01.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:01.955+0000 7fc9477fe640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc934010360 0x7fc93400e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.117:36176/0 (socket says 172.21.15.117:36176) 2024-05-15T12:42:01.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:01.955+0000 7fc9477fe640 1 -- 172.21.15.117:0/1189213620 learned_addr learned my addr 172.21.15.117:0/1189213620 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:42:01.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:01.955+0000 7fc9477fe640 10 monclient(hunting): get_auth_request con 0x7fc934010360 auth_method 0 2024-05-15T12:42:01.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:01.955+0000 7fc9477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:01.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:01.955+0000 7fc9477fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:01.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:01.955+0000 7fc9477fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:01.951 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:01.961+0000 7fc947fff640 1 --2- 172.21.15.117:0/1189213620 >> v2:172.21.15.178:3300/0 conn(0x7fc93400f8e0 0x7fc93400fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:01.952 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:01.962+0000 7fc947fff640 10 monclient(hunting): get_auth_request con 0x7fc93400f8e0 auth_method 0 2024-05-15T12:42:01.952 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:01.962+0000 7fc947fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:01.952 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:01.962+0000 7fc947fff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:01.952 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:01.962+0000 7fc947fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:01.969 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:01.979+0000 7fc947fff640 1 -- 172.21.15.117:0/1189213620 >> v2:172.21.15.178:3300/0 conn(0x7fc93400f8e0 msgr2=0x7fc93400fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-15T12:42:01.969 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:01.979+0000 7fc947fff640 1 -- 172.21.15.117:0/1189213620 >> v2:172.21.15.178:3300/0 conn(0x7fc93400f8e0 msgr2=0x7fc93400fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-15T12:42:01.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:01.979+0000 7fc947fff640 1 --2- 172.21.15.117:0/1189213620 >> v2:172.21.15.178:3300/0 conn(0x7fc93400f8e0 0x7fc93400fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-15T12:42:01.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:01.979+0000 7fc947fff640 1 --2- 172.21.15.117:0/1189213620 >> v2:172.21.15.178:3300/0 conn(0x7fc93400f8e0 0x7fc93400fcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-15T12:42:01.982 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:01.993+0000 7fc9477fe640 1 -- 172.21.15.117:0/1189213620 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc934010360 msgr2=0x7fc93400e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-15T12:42:01.983 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:01.993+0000 7fc9477fe640 1 -- 172.21.15.117:0/1189213620 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc934010360 msgr2=0x7fc93400e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-15T12:42:01.983 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:01.993+0000 7fc9477fe640 1 --2- 172.21.15.117:0/1189213620 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc934010360 0x7fc93400e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-15T12:42:01.983 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:01.993+0000 7fc9477fe640 1 --2- 172.21.15.117:0/1189213620 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc934010360 0x7fc93400e7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-15T12:42:02.442 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:02.452+0000 7fc947fff640 1 --2- 172.21.15.117:0/1189213620 >> v2:172.21.15.178:3300/0 conn(0x7fc93400f8e0 0x7fc93400fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:02.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:02.452+0000 7fc947fff640 10 monclient(hunting): get_auth_request con 0x7fc93400f8e0 auth_method 0 2024-05-15T12:42:02.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:02.452+0000 7fc947fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:02.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:02.452+0000 7fc947fff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:02.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:02.452+0000 7fc947fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-05-15T12:42:02.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:02.453+0000 7fc947fff640 1 -- 172.21.15.117:0/1189213620 >> v2:172.21.15.178:3300/0 conn(0x7fc93400f8e0 msgr2=0x7fc93400fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-15T12:42:02.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:02.453+0000 7fc947fff640 1 -- 172.21.15.117:0/1189213620 >> v2:172.21.15.178:3300/0 conn(0x7fc93400f8e0 msgr2=0x7fc93400fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-15T12:42:02.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:02.453+0000 7fc947fff640 1 --2- 172.21.15.117:0/1189213620 >> v2:172.21.15.178:3300/0 conn(0x7fc93400f8e0 0x7fc93400fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-15T12:42:02.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:02.453+0000 7fc947fff640 1 --2- 172.21.15.117:0/1189213620 >> v2:172.21.15.178:3300/0 conn(0x7fc93400f8e0 0x7fc93400fcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-15T12:42:02.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:02.499+0000 7fc9477fe640 1 --2- 172.21.15.117:0/1189213620 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc934010360 0x7fc93400e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:02.490 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:02.500+0000 7fc9477fe640 10 monclient(hunting): get_auth_request con 0x7fc934010360 auth_method 0 2024-05-15T12:42:02.490 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:02.500+0000 7fc9477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:02.490 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:02.500+0000 7fc9477fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:02.490 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:02.500+0000 7fc9477fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-05-15T12:42:02.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:02.501+0000 7fc9477fe640 1 -- 172.21.15.117:0/1189213620 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc934010360 msgr2=0x7fc93400e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-15T12:42:02.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:02.501+0000 7fc9477fe640 1 -- 172.21.15.117:0/1189213620 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc934010360 msgr2=0x7fc93400e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-15T12:42:02.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:02.501+0000 7fc9477fe640 1 --2- 172.21.15.117:0/1189213620 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc934010360 0x7fc93400e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-15T12:42:02.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:02.501+0000 7fc9477fe640 1 --2- 172.21.15.117:0/1189213620 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc934010360 0x7fc93400e7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-15T12:42:02.850 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:02.860+0000 7fc947fff640 1 --2- 172.21.15.117:0/1189213620 >> v2:172.21.15.178:3300/0 conn(0x7fc93400f8e0 0x7fc93400fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:02.887 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:02.897+0000 7fc947fff640 10 monclient(hunting): get_auth_request con 0x7fc93400f8e0 auth_method 0 2024-05-15T12:42:02.887 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:02.897+0000 7fc947fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:02.887 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:02.897+0000 7fc947fff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:02.887 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:02.897+0000 7fc947fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-05-15T12:42:02.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:02.972+0000 7fc947fff640 1 -- 172.21.15.117:0/1189213620 >> v2:172.21.15.178:3300/0 conn(0x7fc93400f8e0 msgr2=0x7fc93400fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-15T12:42:02.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:02.972+0000 7fc947fff640 1 -- 172.21.15.117:0/1189213620 >> v2:172.21.15.178:3300/0 conn(0x7fc93400f8e0 msgr2=0x7fc93400fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-15T12:42:02.963 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:02.972+0000 7fc947fff640 1 --2- 172.21.15.117:0/1189213620 >> v2:172.21.15.178:3300/0 conn(0x7fc93400f8e0 0x7fc93400fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-15T12:42:02.963 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:02.972+0000 7fc947fff640 1 --2- 172.21.15.117:0/1189213620 >> v2:172.21.15.178:3300/0 conn(0x7fc93400f8e0 0x7fc93400fcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-15T12:42:02.986 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:02.996+0000 7fc9477fe640 1 --2- 172.21.15.117:0/1189213620 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc934010360 0x7fc93400e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:02.987 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:02.997+0000 7fc9477fe640 10 monclient(hunting): get_auth_request con 0x7fc934010360 auth_method 0 2024-05-15T12:42:02.987 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:02.997+0000 7fc9477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:02.987 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:02.997+0000 7fc9477fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:02.987 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:02.997+0000 7fc9477fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-05-15T12:42:02.988 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:02.998+0000 7fc9477fe640 1 -- 172.21.15.117:0/1189213620 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc934010360 msgr2=0x7fc93400e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-15T12:42:02.988 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:02.998+0000 7fc9477fe640 1 -- 172.21.15.117:0/1189213620 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc934010360 msgr2=0x7fc93400e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-15T12:42:02.988 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:02.998+0000 7fc9477fe640 1 --2- 172.21.15.117:0/1189213620 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc934010360 0x7fc93400e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-15T12:42:02.989 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:02.998+0000 7fc9477fe640 1 --2- 172.21.15.117:0/1189213620 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc934010360 0x7fc93400e7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-15T12:42:03.764 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:03.774+0000 7fc947fff640 1 --2- 172.21.15.117:0/1189213620 >> v2:172.21.15.178:3300/0 conn(0x7fc93400f8e0 0x7fc93400fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:03.764 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:03.774+0000 7fc947fff640 10 monclient(hunting): get_auth_request con 0x7fc93400f8e0 auth_method 0 2024-05-15T12:42:03.765 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:03.774+0000 7fc947fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:03.765 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:03.774+0000 7fc947fff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:03.765 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:03.774+0000 7fc947fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-05-15T12:42:03.765 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:03.775+0000 7fc947fff640 1 -- 172.21.15.117:0/1189213620 >> v2:172.21.15.178:3300/0 conn(0x7fc93400f8e0 msgr2=0x7fc93400fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-15T12:42:03.765 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:03.775+0000 7fc947fff640 1 -- 172.21.15.117:0/1189213620 >> v2:172.21.15.178:3300/0 conn(0x7fc93400f8e0 msgr2=0x7fc93400fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-15T12:42:03.765 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:03.775+0000 7fc947fff640 1 --2- 172.21.15.117:0/1189213620 >> v2:172.21.15.178:3300/0 conn(0x7fc93400f8e0 0x7fc93400fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-15T12:42:03.765 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:03.775+0000 7fc947fff640 1 --2- 172.21.15.117:0/1189213620 >> v2:172.21.15.178:3300/0 conn(0x7fc93400f8e0 0x7fc93400fcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-15T12:42:03.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:03.800+0000 7fc9477fe640 1 --2- 172.21.15.117:0/1189213620 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc934010360 0x7fc93400e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:03.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:03.800+0000 7fc9477fe640 10 monclient(hunting): get_auth_request con 0x7fc934010360 auth_method 0 2024-05-15T12:42:03.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:03.800+0000 7fc9477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:03.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:03.800+0000 7fc9477fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:03.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:03.800+0000 7fc9477fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-05-15T12:42:03.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:03.801+0000 7fc9477fe640 1 -- 172.21.15.117:0/1189213620 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc934010360 msgr2=0x7fc93400e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-15T12:42:03.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:03.801+0000 7fc9477fe640 1 -- 172.21.15.117:0/1189213620 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc934010360 msgr2=0x7fc93400e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-15T12:42:03.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:03.801+0000 7fc9477fe640 1 --2- 172.21.15.117:0/1189213620 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc934010360 0x7fc93400e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-15T12:42:03.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:03.801+0000 7fc9477fe640 1 --2- 172.21.15.117:0/1189213620 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc934010360 0x7fc93400e7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-15T12:42:04.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:04.840+0000 7fc945ffb640 10 monclient(hunting): tick 2024-05-15T12:42:04.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:04.840+0000 7fc945ffb640 10 monclient(hunting): _check_auth_tickets 2024-05-15T12:42:04.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:04.840+0000 7fc945ffb640 1 monclient(hunting): continuing hunt 2024-05-15T12:42:04.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:04.840+0000 7fc945ffb640 10 monclient(hunting): _reopen_session rank -1 2024-05-15T12:42:04.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:04.840+0000 7fc945ffb640 1 -- 172.21.15.117:0/1189213620 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc934010360 msgr2=0x7fc93400e7e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-15T12:42:04.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:04.840+0000 7fc945ffb640 1 --2- 172.21.15.117:0/1189213620 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc934010360 0x7fc93400e7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:04.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:04.840+0000 7fc945ffb640 1 -- 172.21.15.117:0/1189213620 >> v2:172.21.15.178:3300/0 conn(0x7fc93400f8e0 msgr2=0x7fc93400fcf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-15T12:42:04.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:04.840+0000 7fc945ffb640 1 --2- 172.21.15.117:0/1189213620 >> v2:172.21.15.178:3300/0 conn(0x7fc93400f8e0 0x7fc93400fcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:04.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:04.840+0000 7fc945ffb640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-05-15T12:42:04.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:04.840+0000 7fc94ca65640 1 -- 172.21.15.117:0/1189213620 reap_dead start 2024-05-15T12:42:04.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:04.840+0000 7fc945ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc93400a3c0 addr v1:172.21.15.178:6789/0 2024-05-15T12:42:04.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:04.840+0000 7fc945ffb640 1 --2- 172.21.15.117:0/1189213620 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc934009f10 0x7fc934009860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:04.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:04.840+0000 7fc945ffb640 10 monclient(hunting): picked mon.noname-b con 0x7fc934009f10 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:42:04.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:04.840+0000 7fc945ffb640 1 --2- 172.21.15.117:0/1189213620 >> v2:172.21.15.178:3300/0 conn(0x7fc93400fcc0 0x7fc93400ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:04.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:04.840+0000 7fc945ffb640 10 monclient(hunting): picked mon.noname-c con 0x7fc93400fcc0 addr v2:172.21.15.178:3300/0 2024-05-15T12:42:04.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:04.840+0000 7fc945ffb640 1 -- 172.21.15.117:0/1189213620 --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc934010880 con 0x7fc93400a3c0 2024-05-15T12:42:04.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:04.840+0000 7fc945ffb640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:04.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:04.840+0000 7fc945ffb640 1 -- 172.21.15.117:0/1189213620 --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9480b23c0 con 0x7fc93400fcc0 2024-05-15T12:42:04.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:04.840+0000 7fc945ffb640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:04.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:04.840+0000 7fc945ffb640 1 -- 172.21.15.117:0/1189213620 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9480b16f0 con 0x7fc934009f10 2024-05-15T12:42:04.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:04.840+0000 7fc945ffb640 10 monclient(hunting): _renew_subs 2024-05-15T12:42:04.863 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:04.873+0000 7fc9477fe640 1 --2- 172.21.15.117:0/1189213620 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc934009f10 0x7fc934009860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:04.868 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:04.878+0000 7fc9477fe640 10 monclient(hunting): get_auth_request con 0x7fc934009f10 auth_method 0 2024-05-15T12:42:04.868 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:04.878+0000 7fc9477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:04.868 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:04.878+0000 7fc9477fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:04.868 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:04.878+0000 7fc9477fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:04.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:04.883+0000 7fc94ca65640 1 --2- 172.21.15.117:0/1189213620 >> v2:172.21.15.178:3300/0 conn(0x7fc93400fcc0 0x7fc93400ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:04.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:04.883+0000 7fc9477fe640 1 -- 172.21.15.117:0/1189213620 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc934009f10 msgr2=0x7fc934009860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-15T12:42:04.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:04.883+0000 7fc9477fe640 1 -- 172.21.15.117:0/1189213620 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc934009f10 msgr2=0x7fc934009860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-15T12:42:04.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:04.883+0000 7fc9477fe640 1 --2- 172.21.15.117:0/1189213620 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc934009f10 0x7fc934009860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-15T12:42:04.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:04.884+0000 7fc9477fe640 1 --2- 172.21.15.117:0/1189213620 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc934009f10 0x7fc934009860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-15T12:42:04.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:04.884+0000 7fc94ca65640 10 monclient(hunting): get_auth_request con 0x7fc93400fcc0 auth_method 0 2024-05-15T12:42:04.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:04.884+0000 7fc94ca65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:04.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:04.884+0000 7fc94ca65640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:04.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:04.884+0000 7fc94ca65640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:04.875 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:04.885+0000 7fc94ca65640 1 -- 172.21.15.117:0/1189213620 >> v2:172.21.15.178:3300/0 conn(0x7fc93400fcc0 msgr2=0x7fc93400ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-15T12:42:04.875 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:04.885+0000 7fc94ca65640 1 -- 172.21.15.117:0/1189213620 >> v2:172.21.15.178:3300/0 conn(0x7fc93400fcc0 msgr2=0x7fc93400ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-15T12:42:04.875 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:04.885+0000 7fc94ca65640 1 --2- 172.21.15.117:0/1189213620 >> v2:172.21.15.178:3300/0 conn(0x7fc93400fcc0 0x7fc93400ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-15T12:42:04.875 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:04.885+0000 7fc94ca65640 1 --2- 172.21.15.117:0/1189213620 >> v2:172.21.15.178:3300/0 conn(0x7fc93400fcc0 0x7fc93400ef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-15T12:42:05.058 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.068+0000 7fc946ffd640 1 -- 172.21.15.117:0/1189213620 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2172252710 0 0) 0x7fc934010880 con 0x7fc93400a3c0 2024-05-15T12:42:05.058 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.068+0000 7fc946ffd640 10 monclient(hunting): _negotiate 2024-05-15T12:42:05.059 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.068+0000 7fc946ffd640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:05.059 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.068+0000 7fc946ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:05.059 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.068+0000 7fc946ffd640 10 monclient(hunting): my global_id is 4099 2024-05-15T12:42:05.059 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.068+0000 7fc946ffd640 1 -- 172.21.15.117:0/1189213620 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc930003520 con 0x7fc93400a3c0 2024-05-15T12:42:05.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.086+0000 7fc9477fe640 1 --2- 172.21.15.117:0/1189213620 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc934009f10 0x7fc934009860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:05.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.087+0000 7fc9477fe640 10 monclient(hunting): get_auth_request con 0x7fc934009f10 auth_method 0 2024-05-15T12:42:05.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.087+0000 7fc9477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:05.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.087+0000 7fc9477fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:05.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.087+0000 7fc9477fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-05-15T12:42:05.080 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.090+0000 7fc94ca65640 1 --2- 172.21.15.117:0/1189213620 >> v2:172.21.15.178:3300/0 conn(0x7fc93400fcc0 0x7fc93400ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:05.081 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.091+0000 7fc94ca65640 10 monclient(hunting): get_auth_request con 0x7fc93400fcc0 auth_method 0 2024-05-15T12:42:05.082 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.091+0000 7fc94ca65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:05.082 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.091+0000 7fc94ca65640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:05.082 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.091+0000 7fc94ca65640 10 monclient(hunting): _init_auth already have auth, reseting 2024-05-15T12:42:05.083 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.093+0000 7fc94ca65640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:05.083 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.093+0000 7fc94ca65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:05.083 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.093+0000 7fc94ca65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:05.083 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.093+0000 7fc9477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:05.083 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.093+0000 7fc9477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:05.084 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.093+0000 7fc9477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:05.123 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.133+0000 7fc94ca65640 10 monclient(hunting): handle_auth_done global_id 4105 payload 274 2024-05-15T12:42:05.123 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.133+0000 7fc94ca65640 1 -- 172.21.15.117:0/1189213620 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc934009f10 msgr2=0x7fc934009860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:42:05.123 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.133+0000 7fc94ca65640 1 --2- 172.21.15.117:0/1189213620 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc934009f10 0x7fc934009860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:05.123 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.133+0000 7fc94ca65640 1 -- 172.21.15.117:0/1189213620 >> v1:172.21.15.178:6789/0 conn(0x7fc93400a3c0 legacy=0x7fc934005ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:05.124 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.133+0000 7fc94ca65640 10 monclient: _finish_hunting 0 2024-05-15T12:42:05.124 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.133+0000 7fc94ca65640 1 monclient: found mon.noname-c 2024-05-15T12:42:05.124 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.133+0000 7fc94ca65640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:42:05.124 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.133+0000 7fc94ca65640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.178:3300/0 2024-05-15T12:42:05.124 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.133+0000 7fc94ca65640 1 -- 172.21.15.117:0/1189213620 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9480ab1b0 con 0x7fc93400fcc0 2024-05-15T12:42:05.145 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.155+0000 7fc94ca65640 1 --2- 172.21.15.117:0/1189213620 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc93400fcc0 0x7fc93400ef70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc948146d20 tx=0x7fc94002c4d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d081d9c2b801495 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:42:05.182 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.192+0000 7fc946ffd640 1 -- 172.21.15.117:0/1189213620 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc940039070 con 0x7fc93400fcc0 2024-05-15T12:42:05.182 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.192+0000 7fc946ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:05.183 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.192+0000 7fc946ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:42:05.183 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.192+0000 7fc946ffd640 10 monclient: dump: 2024-05-15T12:42:05.183 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:42:05.183 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:05.183 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:05.183 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:05.183 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:05.183 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:42:05.183 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:05.183 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:05.183 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:05.183 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:42:05.183 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.192+0000 7fc946ffd640 10 monclient: _finish_auth 0 2024-05-15T12:42:05.183 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.192+0000 7fc946ffd640 10 monclient: _check_auth_tickets 2024-05-15T12:42:05.183 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.192+0000 7fc946ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:42:05.183 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.192+0000 7fc946ffd640 1 -- 172.21.15.117:0/1189213620 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc940033600 con 0x7fc93400fcc0 2024-05-15T12:42:05.183 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.192+0000 7fc94e4c8640 5 monclient: authenticate success, global_id 4105 2024-05-15T12:42:05.184 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.192+0000 7fc94e4c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:42:05.184 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.192+0000 7fc946ffd640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:42:05.184 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.192+0000 7fc94e4c8640 10 monclient: get_monmap_and_config success 2024-05-15T12:42:05.184 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.192+0000 7fc94e4c8640 4 set_mon_vals no callback set 2024-05-15T12:42:05.184 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.192+0000 7fc946ffd640 1 -- 172.21.15.117:0/1189213620 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9400339f0 con 0x7fc93400fcc0 2024-05-15T12:42:05.184 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.192+0000 7fc946ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:05.184 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.192+0000 7fc946ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:42:05.184 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.192+0000 7fc946ffd640 10 monclient: dump: 2024-05-15T12:42:05.184 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:42:05.184 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:05.184 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:05.184 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:05.184 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:05.184 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:42:05.184 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:05.184 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:05.185 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:05.185 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:42:05.187 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.193+0000 7fc94e4c8640 10 monclient: shutdown 2024-05-15T12:42:05.187 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.193+0000 7fc94e4c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:42:05.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.193+0000 7fc94e4c8640 1 -- 172.21.15.117:0/1189213620 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc93400fcc0 msgr2=0x7fc93400ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:05.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.193+0000 7fc94e4c8640 1 --2- 172.21.15.117:0/1189213620 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc93400fcc0 0x7fc93400ef70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc948146d20 tx=0x7fc94002c4d0 comp rx=0 tx=0).stop 2024-05-15T12:42:05.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.193+0000 7fc94e4c8640 1 -- 172.21.15.117:0/1189213620 shutdown_connections 2024-05-15T12:42:05.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.193+0000 7fc94e4c8640 1 --2- 172.21.15.117:0/1189213620 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc93400fcc0 0x7fc93400ef70 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:05.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.193+0000 7fc94e4c8640 1 --2- 172.21.15.117:0/1189213620 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc934009f10 0x7fc934009860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:05.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.193+0000 7fc94e4c8640 1 -- 172.21.15.117:0/1189213620 >> 172.21.15.117:0/1189213620 conn(0x7fc948066cc0 msgr2=0x7fc9480670d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:42:05.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.193+0000 7fc94e4c8640 1 -- 172.21.15.117:0/1189213620 shutdown_connections 2024-05-15T12:42:05.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.193+0000 7fc94e4c8640 1 -- 172.21.15.117:0/1189213620 wait complete. 2024-05-15T12:42:05.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.194+0000 7fc94e4c8640 10 monclient: build_initial_monmap 2024-05-15T12:42:05.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.194+0000 7fc94e4c8640 10 monclient: monmap: 2024-05-15T12:42:05.188 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:42:05.188 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:42:05.188 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-05-15T12:42:05.188 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-05-15T12:42:05.188 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:42:05.188 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:42:05.189 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T12:42:05.189 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T12:42:05.189 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T12:42:05.189 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:42:05.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.195+0000 7fc94e4c8640 1 librados: starting msgr at 2024-05-15T12:42:05.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.195+0000 7fc94e4c8640 1 librados: starting objecter 2024-05-15T12:42:05.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.195+0000 7fc94e4c8640 1 Processor -- start 2024-05-15T12:42:05.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.195+0000 7fc94e4c8640 1 -- start start 2024-05-15T12:42:05.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.195+0000 7fc94e4c8640 1 librados: setting wanted keys 2024-05-15T12:42:05.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.195+0000 7fc94e4c8640 1 librados: calling monclient init 2024-05-15T12:42:05.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.195+0000 7fc94e4c8640 10 monclient: init 2024-05-15T12:42:05.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.196+0000 7fc94e4c8640 10 monclient: _reopen_session rank -1 2024-05-15T12:42:05.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.196+0000 7fc94e4c8640 10 monclient: _add_conns ranks=[1,0,2] 2024-05-15T12:42:05.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.196+0000 7fc94e4c8640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc934009f10 0x7fc948142250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:05.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.196+0000 7fc94e4c8640 10 monclient(hunting): picked mon.noname-b con 0x7fc934009f10 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:42:05.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.196+0000 7fc94e4c8640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc93400a3c0 0x7fc9480ada70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:05.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.196+0000 7fc94e4c8640 10 monclient(hunting): picked mon.noname-a con 0x7fc93400a3c0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T12:42:05.190 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.196+0000 7fc94e4c8640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc93400fcc0 0x7fc9480af9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:05.190 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.196+0000 7fc94e4c8640 10 monclient(hunting): picked mon.noname-c con 0x7fc93400fcc0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:42:05.190 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.196+0000 7fc94e4c8640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:05.190 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.196+0000 7fc94e4c8640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9480b1e80 con 0x7fc93400a3c0 2024-05-15T12:42:05.190 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.196+0000 7fc94e4c8640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:05.190 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.196+0000 7fc94e4c8640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9480a9b50 con 0x7fc934009f10 2024-05-15T12:42:05.190 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.196+0000 7fc94e4c8640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:05.190 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.196+0000 7fc94e4c8640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9480b2000 con 0x7fc93400fcc0 2024-05-15T12:42:05.190 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.196+0000 7fc94e4c8640 10 monclient(hunting): _renew_subs 2024-05-15T12:42:05.190 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.196+0000 7fc94e4c8640 10 monclient(hunting): authenticate will time out at 1288.438433s 2024-05-15T12:42:05.190 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.197+0000 7fc947fff640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc934009f10 0x7fc948142250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:05.190 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.198+0000 7fc947fff640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc934009f10 0x7fc948142250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:38974/0 (socket says 172.21.15.117:38974) 2024-05-15T12:42:05.190 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.198+0000 7fc947fff640 1 -- 172.21.15.117:0/1047022746 learned_addr learned my addr 172.21.15.117:0/1047022746 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:42:05.190 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.198+0000 7fc947fff640 10 monclient(hunting): get_auth_request con 0x7fc934009f10 auth_method 0 2024-05-15T12:42:05.190 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.198+0000 7fc947fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:05.191 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.198+0000 7fc947fff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:05.191 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.198+0000 7fc947fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:05.191 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.198+0000 7fc947fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:05.191 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.198+0000 7fc947fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:05.191 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.198+0000 7fc947fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:05.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.210+0000 7fc947fff640 10 monclient(hunting): handle_auth_done global_id 4111 payload 824 2024-05-15T12:42:05.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.210+0000 7fc947fff640 1 -- 172.21.15.117:0/1047022746 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc93400fcc0 msgr2=0x7fc9480af9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:05.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.210+0000 7fc947fff640 1 --2- 172.21.15.117:0/1047022746 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc93400fcc0 0x7fc9480af9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:05.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.210+0000 7fc947fff640 1 -- 172.21.15.117:0/1047022746 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc93400a3c0 msgr2=0x7fc9480ada70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:05.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.210+0000 7fc947fff640 1 --2- 172.21.15.117:0/1047022746 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc93400a3c0 0x7fc9480ada70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:05.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.210+0000 7fc947fff640 10 monclient: _finish_hunting 0 2024-05-15T12:42:05.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.210+0000 7fc947fff640 1 monclient: found mon.noname-b 2024-05-15T12:42:05.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.210+0000 7fc947fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:42:05.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.210+0000 7fc947fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2024-05-15T12:42:05.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.210+0000 7fc947fff640 1 -- 172.21.15.117:0/1047022746 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9480ab8b0 con 0x7fc934009f10 2024-05-15T12:42:05.202 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.212+0000 7fc947fff640 1 --2- 172.21.15.117:0/1047022746 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc934009f10 0x7fc948142250 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc93800ba30 tx=0x7fc93800bb20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:42:05.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.212+0000 7fc944ff9640 10 client.?.objecter ms_handle_connect 0x7fc934009f10 2024-05-15T12:42:05.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.212+0000 7fc944ff9640 10 client.?.objecter resend_mon_ops 2024-05-15T12:42:05.210 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.220+0000 7fc944ff9640 1 -- 172.21.15.117:0/1047022746 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc938005850 con 0x7fc934009f10 2024-05-15T12:42:05.210 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.220+0000 7fc944ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:05.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.220+0000 7fc944ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:42:05.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.220+0000 7fc944ff9640 10 monclient: dump: 2024-05-15T12:42:05.211 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:42:05.211 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:05.211 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:05.211 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:05.211 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:05.211 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:42:05.211 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:05.211 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:05.211 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:05.211 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:42:05.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.220+0000 7fc944ff9640 10 monclient: _finish_auth 0 2024-05-15T12:42:05.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.220+0000 7fc944ff9640 10 monclient: _check_auth_tickets 2024-05-15T12:42:05.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.220+0000 7fc944ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:42:05.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.220+0000 7fc944ff9640 1 -- 172.21.15.117:0/1047022746 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc93801b070 con 0x7fc934009f10 2024-05-15T12:42:05.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.220+0000 7fc94e4c8640 5 monclient: authenticate success, global_id 4111 2024-05-15T12:42:05.212 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.220+0000 7fc944ff9640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:42:05.212 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.220+0000 7fc944ff9640 1 -- 172.21.15.117:0/1047022746 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9380163f0 con 0x7fc934009f10 2024-05-15T12:42:05.212 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.220+0000 7fc944ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:05.212 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.220+0000 7fc944ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:42:05.212 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.220+0000 7fc944ff9640 10 monclient: dump: 2024-05-15T12:42:05.212 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:42:05.212 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:05.212 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:05.212 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:05.212 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:05.212 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:42:05.212 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:05.212 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:05.212 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:05.212 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:42:05.212 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.220+0000 7fc94e4c8640 10 monclient: _renew_subs 2024-05-15T12:42:05.212 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.220+0000 7fc94e4c8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:42:05.213 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.220+0000 7fc94e4c8640 1 -- 172.21.15.117:0/1047022746 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9480ad2d0 con 0x7fc934009f10 2024-05-15T12:42:05.213 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.220+0000 7fc945ffb640 4 set_mon_vals no callback set 2024-05-15T12:42:05.213 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.220+0000 7fc94e4c8640 10 client.4111.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:42:05.213 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.220+0000 7fc94e4c8640 10 monclient: _renew_subs 2024-05-15T12:42:05.213 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.220+0000 7fc94e4c8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:42:05.213 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.220+0000 7fc94e4c8640 1 -- 172.21.15.117:0/1047022746 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9480b0840 con 0x7fc934009f10 2024-05-15T12:42:05.213 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.220+0000 7fc94e4c8640 1 librados: init done 2024-05-15T12:42:05.214 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.221+0000 7fc94e4c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:42:05.214 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.221+0000 7fc94e4c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:42:05.214 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.221+0000 7fc94e4c8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:42:05.214 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.221+0000 7fc94e4c8640 1 -- 172.21.15.117:0/1047022746 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc94804f0e0 con 0x7fc934009f10 2024-05-15T12:42:05.384 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-05-15T12:42:05.394+0000 7f5134c38040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-15T12:42:05.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.412+0000 7fc944ff9640 1 -- 172.21.15.117:0/1047022746 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fc938016590 con 0x7fc934009f10 2024-05-15T12:42:05.409 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.419+0000 7fc944ff9640 1 -- 172.21.15.117:0/1047022746 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fc9380209e0 con 0x7fc934009f10 2024-05-15T12:42:05.409 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.419+0000 7fc944ff9640 10 client.4111.objecter ms_dispatch 0x7fc948000b90 osd_map(1..1 src has 1..1) v4 2024-05-15T12:42:05.409 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.419+0000 7fc944ff9640 3 client.4111.objecter handle_osd_map got epochs [1,1] > 0 2024-05-15T12:42:05.410 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.419+0000 7fc944ff9640 3 client.4111.objecter handle_osd_map decoding full epoch 1 2024-05-15T12:42:05.410 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.419+0000 7fc944ff9640 20 client.4111.objecter dump_active .. 0 homeless 2024-05-15T12:42:05.410 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.419+0000 7fc944ff9640 1 -- 172.21.15.117:0/1047022746 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fc9480b0840 con 0x7fc934009f10 2024-05-15T12:42:05.410 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.419+0000 7fc944ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:42:05.410 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.419+0000 7fc944ff9640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:42:05.474 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.484+0000 7fc94e4c8640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-05-15T12:42:05.474 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.484+0000 7fc94e4c8640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-05-15T12:42:05.475 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.484+0000 7fc94e4c8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:42:05.475 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.484+0000 7fc94e4c8640 1 -- 172.21.15.117:0/1047022746 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fc948067d90 con 0x7fc934009f10 2024-05-15T12:42:05.558 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-05-15T12:42:05.568+0000 7f5134c38040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-15T12:42:05.610 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-05-15T12:42:05.620+0000 7f18296bb040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-15T12:42:05.611 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-05-15T12:42:05.622+0000 7f5134c38040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-15T12:42:05.774 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.783+0000 7fc944ff9640 1 -- 172.21.15.117:0/1047022746 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7fc938013070 con 0x7fc934009f10 2024-05-15T12:42:05.774 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.783+0000 7fc944ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-05-15T12:42:05.774 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.783+0000 7fc944ff9640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-05-15T12:42:05.774 INFO:teuthology.orchestra.run.smithi117.stderr:adjusted tunables profile to default 2024-05-15T12:42:05.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.784+0000 7fc94e4c8640 10 librados: watch_flush enter 2024-05-15T12:42:05.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.784+0000 7fc94e4c8640 10 librados: watch_flush exit 2024-05-15T12:42:05.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.784+0000 7fc94e4c8640 20 client.4111.objecter shutdown clearing up homeless session... 2024-05-15T12:42:05.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.784+0000 7fc94e4c8640 10 client.4111.objecter successfully canceled tick 2024-05-15T12:42:05.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.784+0000 7fc94e4c8640 10 monclient: shutdown 2024-05-15T12:42:05.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.784+0000 7fc94e4c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:42:05.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.784+0000 7fc94e4c8640 1 -- 172.21.15.117:0/1047022746 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc934009f10 msgr2=0x7fc948142250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:05.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.784+0000 7fc94e4c8640 1 --2- 172.21.15.117:0/1047022746 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc934009f10 0x7fc948142250 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc93800ba30 tx=0x7fc93800bb20 comp rx=0 tx=0).stop 2024-05-15T12:42:05.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.784+0000 7fc94e4c8640 1 -- 172.21.15.117:0/1047022746 shutdown_connections 2024-05-15T12:42:05.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.784+0000 7fc94e4c8640 1 --2- 172.21.15.117:0/1047022746 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc93400fcc0 0x7fc9480af9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:05.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.784+0000 7fc94e4c8640 1 --2- 172.21.15.117:0/1047022746 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc93400a3c0 0x7fc9480ada70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:05.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.784+0000 7fc94e4c8640 1 --2- 172.21.15.117:0/1047022746 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc934009f10 0x7fc948142250 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:05.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.784+0000 7fc94e4c8640 1 -- 172.21.15.117:0/1047022746 >> 172.21.15.117:0/1047022746 conn(0x7fc948066cc0 msgr2=0x7fc9480b3780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:42:05.777 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.784+0000 7fc94e4c8640 1 -- 172.21.15.117:0/1047022746 shutdown_connections 2024-05-15T12:42:05.777 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.784+0000 7fc94e4c8640 1 -- 172.21.15.117:0/1047022746 wait complete. 2024-05-15T12:42:05.777 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:05.784+0000 7fc94e4c8640 1 librados: shutdown 2024-05-15T12:42:05.785 INFO:tasks.ceph:check_enable_crimson: False 2024-05-15T12:42:05.785 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-05-15T12:42:05.786 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-05-15T12:42:05.786 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-05-15T12:42:05.790 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-05-15T12:42:05.800+0000 7f18296bb040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-15T12:42:05.811 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-05-15T12:42:05.811 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-05-15T12:42:05.842 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-05-15T12:42:05.852+0000 7f18296bb040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-15T12:42:05.875 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-05-15T12:42:05.875 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-05-15T12:42:05.905 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-05-15T12:42:05.914+0000 7f5134c38040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-15T12:42:05.929 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-05-15T12:42:05.930 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-05-15T12:42:05.961 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-05-15T12:42:05.971+0000 7f5134c38040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-15T12:42:05.985 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-05-15T12:42:05.985 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-05-15T12:42:06.010 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-05-15T12:42:06.010 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-05-15T12:42:06.076 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-05-15T12:42:06.076 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-05-15T12:42:06.132 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-05-15T12:42:06.142+0000 7f18296bb040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-15T12:42:06.141 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-05-15T12:42:06.141 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-05-15T12:42:06.180 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-05-15T12:42:06.190+0000 7f18296bb040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-15T12:42:06.204 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph --cluster ceph osd new aaffd6d9-1a00-4793-9d1f-65a30501bec4 0 2024-05-15T12:42:06.363 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.373+0000 7fa13626a640 10 monclient: get_monmap_and_config 2024-05-15T12:42:06.363 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.373+0000 7fa13626a640 10 monclient: build_initial_monmap 2024-05-15T12:42:06.363 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.373+0000 7fa13626a640 10 monclient: monmap: 2024-05-15T12:42:06.363 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 0 2024-05-15T12:42:06.363 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:06.363 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-05-15T12:42:06.374471+0000 2024-05-15T12:42:06.363 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-05-15T12:42:06.374471+0000 2024-05-15T12:42:06.364 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 0 (unknown) 2024-05-15T12:42:06.364 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2024-05-15T12:42:06.364 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T12:42:06.364 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T12:42:06.364 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T12:42:06.364 INFO:teuthology.orchestra.run.smithi178.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T12:42:06.364 INFO:teuthology.orchestra.run.smithi178.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T12:42:06.364 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-05-15T12:42:06.366 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.374+0000 7fa13626a640 1 Processor -- start 2024-05-15T12:42:06.366 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.374+0000 7fa13626a640 1 -- start start 2024-05-15T12:42:06.366 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.374+0000 7fa13626a640 10 monclient: init 2024-05-15T12:42:06.366 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.374+0000 7fa13626a640 10 monclient: _reopen_session rank -1 2024-05-15T12:42:06.366 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.374+0000 7fa13626a640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-05-15T12:42:06.366 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.374+0000 7fa13626a640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fa130070bc0 0x7fa130144190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:06.366 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.374+0000 7fa13626a640 10 monclient(hunting): picked mon.noname-a con 0x7fa130070bc0 addr v2:172.21.15.117:3300/0 2024-05-15T12:42:06.366 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.374+0000 7fa13626a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa1301446d0 addr v1:172.21.15.117:6789/0 2024-05-15T12:42:06.367 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.374+0000 7fa13626a640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fa1300b07e0 0x7fa13014ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:06.367 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.374+0000 7fa13626a640 10 monclient(hunting): picked mon.noname-c con 0x7fa1300b07e0 addr v2:172.21.15.178:3300/0 2024-05-15T12:42:06.367 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.374+0000 7fa13626a640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa130072b30 con 0x7fa1301446d0 2024-05-15T12:42:06.367 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.374+0000 7fa13626a640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:06.367 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.374+0000 7fa13626a640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1300a9550 con 0x7fa130070bc0 2024-05-15T12:42:06.367 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.374+0000 7fa13626a640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:06.367 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.374+0000 7fa13626a640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa130071fa0 con 0x7fa1300b07e0 2024-05-15T12:42:06.367 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.374+0000 7fa13626a640 10 monclient(hunting): _renew_subs 2024-05-15T12:42:06.367 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.375+0000 7fa13626a640 10 monclient(hunting): authenticate will time out at 1282.712386s 2024-05-15T12:42:06.367 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.375+0000 7fa134807640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fa1300b07e0 0x7fa13014ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:06.367 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.375+0000 7fa12ffff640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fa130070bc0 0x7fa130144190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:06.367 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.375+0000 7fa134807640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fa1300b07e0 0x7fa13014ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.178:50424/0 (socket says 172.21.15.178:50424) 2024-05-15T12:42:06.367 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.375+0000 7fa134807640 1 -- 172.21.15.178:0/1274487377 learned_addr learned my addr 172.21.15.178:0/1274487377 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-05-15T12:42:06.367 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.375+0000 7fa134807640 10 monclient(hunting): get_auth_request con 0x7fa1300b07e0 auth_method 0 2024-05-15T12:42:06.367 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.375+0000 7fa134807640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:06.367 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.375+0000 7fa134807640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:06.367 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.375+0000 7fa134807640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:06.368 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.375+0000 7fa12ffff640 10 monclient(hunting): get_auth_request con 0x7fa130070bc0 auth_method 0 2024-05-15T12:42:06.368 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.375+0000 7fa12ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:06.368 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.375+0000 7fa12ffff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:06.368 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.375+0000 7fa12ffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:06.368 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.376+0000 7fa134807640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:06.368 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.376+0000 7fa134807640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:06.368 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.376+0000 7fa134807640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:06.369 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.376+0000 7fa12ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:06.369 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.376+0000 7fa12ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:06.369 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.376+0000 7fa12ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:06.369 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.378+0000 7fa134807640 10 monclient(hunting): handle_auth_done global_id 4117 payload 274 2024-05-15T12:42:06.369 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.378+0000 7fa134807640 1 -- 172.21.15.178:0/1274487377 >> v2:172.21.15.117:3300/0 conn(0x7fa130070bc0 msgr2=0x7fa130144190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:42:06.369 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.378+0000 7fa134807640 1 --2- 172.21.15.178:0/1274487377 >> v2:172.21.15.117:3300/0 conn(0x7fa130070bc0 0x7fa130144190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:06.369 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.378+0000 7fa134807640 1 -- 172.21.15.178:0/1274487377 >> v1:172.21.15.117:6789/0 conn(0x7fa1301446d0 legacy=0x7fa13013d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:06.369 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.378+0000 7fa134807640 10 monclient: _finish_hunting 0 2024-05-15T12:42:06.369 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.378+0000 7fa134807640 1 monclient: found mon.noname-c 2024-05-15T12:42:06.369 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.378+0000 7fa134807640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:42:06.370 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.378+0000 7fa134807640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.178:3300/0 2024-05-15T12:42:06.370 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.378+0000 7fa134807640 1 -- 172.21.15.178:0/1274487377 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa13013dff0 con 0x7fa1300b07e0 2024-05-15T12:42:06.370 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.380+0000 7fa134807640 1 --2- 172.21.15.178:0/1274487377 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa1300b07e0 0x7fa13014ae60 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa1300b2750 tx=0x7fa12802bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=950b6f8fef67540d server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:42:06.374 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.384+0000 7fa12effd640 1 -- 172.21.15.178:0/1274487377 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa12802ca80 con 0x7fa1300b07e0 2024-05-15T12:42:06.374 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.384+0000 7fa12effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:06.374 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.384+0000 7fa12effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:42:06.374 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.384+0000 7fa12effd640 10 monclient: dump: 2024-05-15T12:42:06.374 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-05-15T12:42:06.374 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:06.375 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:06.375 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:06.375 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:06.375 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2024-05-15T12:42:06.375 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:06.375 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:06.375 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:06.375 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-05-15T12:42:06.375 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.384+0000 7fa12effd640 10 monclient: _finish_auth 0 2024-05-15T12:42:06.375 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.384+0000 7fa12effd640 10 monclient: _check_auth_tickets 2024-05-15T12:42:06.375 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.384+0000 7fa12effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:42:06.375 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.384+0000 7fa12effd640 1 -- 172.21.15.178:0/1274487377 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa12802cc20 con 0x7fa1300b07e0 2024-05-15T12:42:06.375 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.384+0000 7fa12effd640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:42:06.375 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.384+0000 7fa12effd640 1 -- 172.21.15.178:0/1274487377 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1280343f0 con 0x7fa1300b07e0 2024-05-15T12:42:06.375 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.384+0000 7fa12effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:06.375 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.384+0000 7fa12effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:42:06.375 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.384+0000 7fa12effd640 10 monclient: dump: 2024-05-15T12:42:06.376 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-05-15T12:42:06.376 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:06.376 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:06.376 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:06.376 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:06.376 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2024-05-15T12:42:06.376 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:06.376 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:06.376 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:06.376 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-05-15T12:42:06.376 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.384+0000 7fa13626a640 5 monclient: authenticate success, global_id 4117 2024-05-15T12:42:06.376 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.384+0000 7fa13626a640 10 monclient: get_monmap_and_config success 2024-05-15T12:42:06.376 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.384+0000 7fa13626a640 4 set_mon_vals no callback set 2024-05-15T12:42:06.380 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.385+0000 7fa13626a640 10 monclient: shutdown 2024-05-15T12:42:06.380 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.385+0000 7fa13626a640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:42:06.380 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.385+0000 7fa13626a640 1 -- 172.21.15.178:0/1274487377 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa1300b07e0 msgr2=0x7fa13014ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:06.380 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.385+0000 7fa13626a640 1 --2- 172.21.15.178:0/1274487377 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa1300b07e0 0x7fa13014ae60 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa1300b2750 tx=0x7fa12802bee0 comp rx=0 tx=0).stop 2024-05-15T12:42:06.380 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.385+0000 7fa13626a640 1 -- 172.21.15.178:0/1274487377 shutdown_connections 2024-05-15T12:42:06.380 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.385+0000 7fa13626a640 1 --2- 172.21.15.178:0/1274487377 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa1300b07e0 0x7fa13014ae60 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:06.380 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.385+0000 7fa13626a640 1 --2- 172.21.15.178:0/1274487377 >> v2:172.21.15.117:3300/0 conn(0x7fa130070bc0 0x7fa130144190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:06.380 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.385+0000 7fa13626a640 1 -- 172.21.15.178:0/1274487377 >> 172.21.15.178:0/1274487377 conn(0x7fa1300b2ac0 msgr2=0x7fa1300b2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:42:06.380 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.385+0000 7fa13626a640 1 -- 172.21.15.178:0/1274487377 shutdown_connections 2024-05-15T12:42:06.380 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.385+0000 7fa13626a640 1 -- 172.21.15.178:0/1274487377 wait complete. 2024-05-15T12:42:06.381 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.385+0000 7fa13626a640 10 monclient: build_initial_monmap 2024-05-15T12:42:06.381 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.385+0000 7fa13626a640 10 monclient: monmap: 2024-05-15T12:42:06.381 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 0 2024-05-15T12:42:06.381 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:42:06.381 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 0.000000 2024-05-15T12:42:06.381 INFO:teuthology.orchestra.run.smithi178.stderr:created 0.000000 2024-05-15T12:42:06.381 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 0 (unknown) 2024-05-15T12:42:06.381 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2024-05-15T12:42:06.381 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T12:42:06.381 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T12:42:06.381 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T12:42:06.381 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-05-15T12:42:06.381 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.386+0000 7fa13626a640 1 librados: starting msgr at 2024-05-15T12:42:06.381 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.386+0000 7fa13626a640 1 librados: starting objecter 2024-05-15T12:42:06.381 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.386+0000 7fa13626a640 1 Processor -- start 2024-05-15T12:42:06.381 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.386+0000 7fa13626a640 1 -- start start 2024-05-15T12:42:06.381 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.386+0000 7fa13626a640 1 librados: setting wanted keys 2024-05-15T12:42:06.382 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.386+0000 7fa13626a640 1 librados: calling monclient init 2024-05-15T12:42:06.382 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.386+0000 7fa13626a640 10 monclient: init 2024-05-15T12:42:06.382 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.387+0000 7fa13626a640 10 monclient: _reopen_session rank -1 2024-05-15T12:42:06.382 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.387+0000 7fa13626a640 10 monclient: _add_conns ranks=[0,1,2] 2024-05-15T12:42:06.382 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.387+0000 7fa13626a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa130070bc0 0x7fa130145300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:06.382 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.387+0000 7fa13626a640 10 monclient(hunting): picked mon.noname-a con 0x7fa130070bc0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T12:42:06.382 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.387+0000 7fa13626a640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa1300b07e0 0x7fa130145d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:06.382 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.387+0000 7fa13626a640 10 monclient(hunting): picked mon.noname-b con 0x7fa1300b07e0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:42:06.382 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.387+0000 7fa13626a640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa1301d5e20 0x7fa1301d62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:06.382 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.387+0000 7fa13626a640 10 monclient(hunting): picked mon.noname-c con 0x7fa1301d5e20 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:42:06.382 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.387+0000 7fa13626a640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:06.382 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.387+0000 7fa13626a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa13014b3a0 con 0x7fa130070bc0 2024-05-15T12:42:06.382 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.387+0000 7fa13626a640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:06.382 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.387+0000 7fa13626a640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa130071230 con 0x7fa1300b07e0 2024-05-15T12:42:06.383 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.387+0000 7fa13626a640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:06.383 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.387+0000 7fa13626a640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa13014b520 con 0x7fa1301d5e20 2024-05-15T12:42:06.383 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.387+0000 7fa13626a640 10 monclient(hunting): _renew_subs 2024-05-15T12:42:06.383 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.387+0000 7fa13626a640 10 monclient(hunting): authenticate will time out at 1282.724534s 2024-05-15T12:42:06.383 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.387+0000 7fa12f7fe640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa1300b07e0 0x7fa130145d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:06.383 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.387+0000 7fa12ffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa130070bc0 0x7fa130145300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:06.383 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.387+0000 7fa12f7fe640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa1300b07e0 0x7fa130145d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.178:50430/0 (socket says 172.21.15.178:50430) 2024-05-15T12:42:06.383 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.387+0000 7fa12f7fe640 1 -- 172.21.15.178:0/778828130 learned_addr learned my addr 172.21.15.178:0/778828130 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-05-15T12:42:06.383 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.387+0000 7fa12f7fe640 10 monclient(hunting): get_auth_request con 0x7fa1300b07e0 auth_method 0 2024-05-15T12:42:06.383 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.387+0000 7fa12f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:06.383 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.387+0000 7fa12f7fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:06.383 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.387+0000 7fa12f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:06.383 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.388+0000 7fa134807640 1 --2- 172.21.15.178:0/778828130 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa1301d5e20 0x7fa1301d62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:06.383 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.388+0000 7fa12ffff640 10 monclient(hunting): get_auth_request con 0x7fa130070bc0 auth_method 0 2024-05-15T12:42:06.383 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.388+0000 7fa12ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:06.383 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.388+0000 7fa12ffff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:06.383 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.388+0000 7fa12ffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:06.384 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.388+0000 7fa134807640 10 monclient(hunting): get_auth_request con 0x7fa1301d5e20 auth_method 0 2024-05-15T12:42:06.384 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.388+0000 7fa134807640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:06.384 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.388+0000 7fa134807640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:06.384 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.388+0000 7fa134807640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:06.384 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.388+0000 7fa12f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:06.384 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.388+0000 7fa12f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:06.384 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.388+0000 7fa12f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:06.384 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.388+0000 7fa12ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:06.384 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.388+0000 7fa12ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:06.384 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.388+0000 7fa12ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:06.384 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.389+0000 7fa134807640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:06.384 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.389+0000 7fa134807640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:06.384 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.389+0000 7fa134807640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:06.385 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.391+0000 7fa12f7fe640 10 monclient(hunting): handle_auth_done global_id 4120 payload 824 2024-05-15T12:42:06.385 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.391+0000 7fa12f7fe640 1 -- 172.21.15.178:0/778828130 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa1301d5e20 msgr2=0x7fa1301d62b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:06.385 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.391+0000 7fa12f7fe640 1 --2- 172.21.15.178:0/778828130 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa1301d5e20 0x7fa1301d62b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:06.386 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.391+0000 7fa12f7fe640 1 -- 172.21.15.178:0/778828130 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa130070bc0 msgr2=0x7fa130145300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:06.386 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.391+0000 7fa12f7fe640 1 --2- 172.21.15.178:0/778828130 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa130070bc0 0x7fa130145300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:06.386 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.391+0000 7fa12f7fe640 10 monclient: _finish_hunting 0 2024-05-15T12:42:06.386 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.391+0000 7fa12f7fe640 1 monclient: found mon.noname-b 2024-05-15T12:42:06.386 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.391+0000 7fa12f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:42:06.386 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.391+0000 7fa12f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2024-05-15T12:42:06.386 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.391+0000 7fa12f7fe640 1 -- 172.21.15.178:0/778828130 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa130147440 con 0x7fa1300b07e0 2024-05-15T12:42:06.386 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.393+0000 7fa12f7fe640 1 --2- 172.21.15.178:0/778828130 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa1300b07e0 0x7fa130145d80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa12400c3c0 tx=0x7fa12400c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:42:06.386 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.393+0000 7fa12cff9640 10 client.?.objecter ms_handle_connect 0x7fa1300b07e0 2024-05-15T12:42:06.386 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.393+0000 7fa12cff9640 10 client.?.objecter resend_mon_ops 2024-05-15T12:42:06.389 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.398+0000 7fa12f7fe640 1 -- 172.21.15.178:0/778828130 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa1300b07e0 msgr2=0x7fa130145d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-05-15T12:42:06.389 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.398+0000 7fa12f7fe640 1 -- 172.21.15.178:0/778828130 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa1300b07e0 msgr2=0x7fa130145d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-15T12:42:06.389 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.398+0000 7fa12f7fe640 1 --2- 172.21.15.178:0/778828130 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa1300b07e0 0x7fa130145d80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa12400c3c0 tx=0x7fa12400c880 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-15T12:42:06.389 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.398+0000 7fa12f7fe640 1 --2- 172.21.15.178:0/778828130 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa1300b07e0 0x7fa130145d80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa12400c3c0 tx=0x7fa12400c880 comp rx=0 tx=0).stop 2024-05-15T12:42:06.389 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.398+0000 7fa12cff9640 10 monclient: ms_handle_reset current mon [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:42:06.389 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.398+0000 7fa12cff9640 10 monclient: _reopen_session rank -1 2024-05-15T12:42:06.389 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.398+0000 7fa12cff9640 1 -- 172.21.15.178:0/778828130 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa1300b07e0 msgr2=0x7fa130145d80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-15T12:42:06.389 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.398+0000 7fa12cff9640 1 --2- 172.21.15.178:0/778828130 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa1300b07e0 0x7fa130145d80 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:06.389 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.398+0000 7fa12cff9640 10 monclient: _add_conns ranks=[1,2,0] 2024-05-15T12:42:06.389 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.398+0000 7fa12cff9640 1 --2- 172.21.15.178:0/778828130 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa11c004e30 0x7fa11c007280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:06.389 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.398+0000 7fa12cff9640 10 monclient(hunting): picked mon.noname-b con 0x7fa11c004e30 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:42:06.390 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.398+0000 7fa12cff9640 1 --2- 172.21.15.178:0/778828130 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa1301d5e20 0x7fa11c009df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:06.390 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.398+0000 7fa12cff9640 10 monclient(hunting): picked mon.noname-c con 0x7fa1301d5e20 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:42:06.390 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.398+0000 7fa12cff9640 1 --2- 172.21.15.178:0/778828130 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa130070bc0 0x7fa11c00c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:06.390 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.398+0000 7fa12cff9640 10 monclient(hunting): picked mon.noname-a con 0x7fa130070bc0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T12:42:06.390 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.398+0000 7fa12cff9640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:06.390 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.398+0000 7fa12cff9640 1 -- 172.21.15.178:0/778828130 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa11c00d320 con 0x7fa130070bc0 2024-05-15T12:42:06.390 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.398+0000 7fa12cff9640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:06.390 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.398+0000 7fa12cff9640 1 -- 172.21.15.178:0/778828130 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa11c00d4d0 con 0x7fa11c004e30 2024-05-15T12:42:06.390 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.398+0000 7fa12cff9640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:06.390 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.398+0000 7fa12cff9640 1 -- 172.21.15.178:0/778828130 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa11c00d6d0 con 0x7fa1301d5e20 2024-05-15T12:42:06.390 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.398+0000 7fa12cff9640 10 monclient(hunting): _renew_subs 2024-05-15T12:42:06.390 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.398+0000 7fa134807640 1 --2- 172.21.15.178:0/778828130 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa130070bc0 0x7fa11c00c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:06.390 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.399+0000 7fa134807640 10 monclient(hunting): get_auth_request con 0x7fa130070bc0 auth_method 0 2024-05-15T12:42:06.390 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.399+0000 7fa134807640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:06.390 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.399+0000 7fa134807640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:06.390 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.399+0000 7fa134807640 10 monclient(hunting): _init_auth already have auth, reseting 2024-05-15T12:42:06.391 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.399+0000 7fa134807640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:06.391 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.399+0000 7fa134807640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:06.391 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.399+0000 7fa134807640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-05-15T12:42:06.393 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.403+0000 7fa12ffff640 1 --2- 172.21.15.178:0/778828130 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa11c004e30 0x7fa11c007280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:06.394 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.404+0000 7fa12ffff640 10 monclient(hunting): get_auth_request con 0x7fa11c004e30 auth_method 0 2024-05-15T12:42:06.394 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.404+0000 7fa12ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:06.394 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.404+0000 7fa12ffff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:06.394 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.404+0000 7fa12ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-05-15T12:42:06.394 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.404+0000 7fa12ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:06.394 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.404+0000 7fa12ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:06.394 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.404+0000 7fa12ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-05-15T12:42:06.396 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.406+0000 7fa12f7fe640 1 --2- 172.21.15.178:0/778828130 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa1301d5e20 0x7fa11c009df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:06.397 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.407+0000 7fa12f7fe640 10 monclient(hunting): get_auth_request con 0x7fa1301d5e20 auth_method 0 2024-05-15T12:42:06.397 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.407+0000 7fa12f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:06.397 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.407+0000 7fa12f7fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:06.397 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.407+0000 7fa12f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-05-15T12:42:06.397 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.407+0000 7fa12f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:06.397 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.407+0000 7fa12f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:06.397 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.408+0000 7fa12f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-05-15T12:42:06.400 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.410+0000 7fa12f7fe640 10 monclient(hunting): handle_auth_done global_id 4120 payload 293 2024-05-15T12:42:06.400 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.410+0000 7fa12f7fe640 1 -- 172.21.15.178:0/778828130 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa11c004e30 msgr2=0x7fa11c007280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:06.400 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.410+0000 7fa12f7fe640 1 --2- 172.21.15.178:0/778828130 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa11c004e30 0x7fa11c007280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:06.400 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.410+0000 7fa12f7fe640 1 -- 172.21.15.178:0/778828130 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa130070bc0 msgr2=0x7fa11c00c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:06.400 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.410+0000 7fa12f7fe640 1 --2- 172.21.15.178:0/778828130 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa130070bc0 0x7fa11c00c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:06.401 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.410+0000 7fa12f7fe640 10 monclient: _finish_hunting 0 2024-05-15T12:42:06.401 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.410+0000 7fa12f7fe640 1 monclient: found mon.noname-c 2024-05-15T12:42:06.401 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.410+0000 7fa12f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:42:06.401 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.410+0000 7fa12f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.117:3301/0 2024-05-15T12:42:06.401 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.410+0000 7fa12f7fe640 1 -- 172.21.15.178:0/778828130 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa11c00d930 con 0x7fa1301d5e20 2024-05-15T12:42:06.402 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.413+0000 7fa12f7fe640 1 --2- 172.21.15.178:0/778828130 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa1301d5e20 0x7fa11c009df0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa1300b4320 tx=0x7fa124007f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:42:06.402 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.413+0000 7fa12cff9640 10 client.?.objecter ms_handle_connect 0x7fa1301d5e20 2024-05-15T12:42:06.402 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.413+0000 7fa12cff9640 10 client.?.objecter resend_mon_ops 2024-05-15T12:42:06.408 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.417+0000 7fa12cff9640 1 -- 172.21.15.178:0/778828130 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa124005300 con 0x7fa1301d5e20 2024-05-15T12:42:06.408 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.417+0000 7fa12cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:06.408 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.417+0000 7fa12cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:42:06.408 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.417+0000 7fa12cff9640 10 monclient: dump: 2024-05-15T12:42:06.408 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-05-15T12:42:06.408 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:06.408 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:06.408 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:06.409 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:06.409 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2024-05-15T12:42:06.409 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:06.409 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:06.409 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:06.409 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-05-15T12:42:06.409 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.417+0000 7fa12cff9640 10 monclient: _finish_auth 0 2024-05-15T12:42:06.409 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.417+0000 7fa12cff9640 10 monclient: _check_auth_tickets 2024-05-15T12:42:06.409 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.417+0000 7fa12cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:42:06.409 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.417+0000 7fa12cff9640 1 -- 172.21.15.178:0/778828130 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1240037f0 con 0x7fa1301d5e20 2024-05-15T12:42:06.409 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.417+0000 7fa12cff9640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:42:06.409 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.417+0000 7fa12cff9640 1 -- 172.21.15.178:0/778828130 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa124003bc0 con 0x7fa1301d5e20 2024-05-15T12:42:06.409 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.417+0000 7fa13626a640 5 monclient: authenticate success, global_id 4120 2024-05-15T12:42:06.409 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.417+0000 7fa13626a640 10 monclient: _renew_subs 2024-05-15T12:42:06.409 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.417+0000 7fa13626a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T12:42:06.409 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.417+0000 7fa13626a640 1 -- 172.21.15.178:0/778828130 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1301d5900 con 0x7fa1301d5e20 2024-05-15T12:42:06.410 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.417+0000 7fa12cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:06.410 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.417+0000 7fa12cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T12:42:06.410 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.417+0000 7fa12cff9640 10 monclient: dump: 2024-05-15T12:42:06.410 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-05-15T12:42:06.410 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:06.410 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:06.410 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:06.410 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:06.410 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2024-05-15T12:42:06.410 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:06.410 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:06.410 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:06.410 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-05-15T12:42:06.410 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.417+0000 7fa12e7fc640 4 set_mon_vals no callback set 2024-05-15T12:42:06.410 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.417+0000 7fa13626a640 10 client.4120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:42:06.410 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.417+0000 7fa13626a640 10 monclient: _renew_subs 2024-05-15T12:42:06.411 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.417+0000 7fa13626a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T12:42:06.411 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.417+0000 7fa13626a640 1 -- 172.21.15.178:0/778828130 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa13013ed00 con 0x7fa1301d5e20 2024-05-15T12:42:06.411 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.417+0000 7fa13626a640 1 librados: init done 2024-05-15T12:42:06.411 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.418+0000 7fa13626a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:42:06.411 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.418+0000 7fa13626a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:42:06.411 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.418+0000 7fa13626a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T12:42:06.411 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.418+0000 7fa13626a640 1 -- 172.21.15.178:0/778828130 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa13004f0e0 con 0x7fa1301d5e20 2024-05-15T12:42:06.429 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.439+0000 7fa12cff9640 1 -- 172.21.15.178:0/778828130 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fa12401b040 con 0x7fa1301d5e20 2024-05-15T12:42:06.429 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.439+0000 7fa12cff9640 1 -- 172.21.15.178:0/778828130 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa12401d850 con 0x7fa1301d5e20 2024-05-15T12:42:06.429 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.439+0000 7fa12cff9640 10 client.4120.objecter ms_dispatch 0x7fa130000b90 osd_map(2..2 src has 1..2) v4 2024-05-15T12:42:06.429 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.439+0000 7fa12cff9640 3 client.4120.objecter handle_osd_map got epochs [2,2] > 0 2024-05-15T12:42:06.429 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.439+0000 7fa12cff9640 3 client.4120.objecter handle_osd_map decoding full epoch 2 2024-05-15T12:42:06.429 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.439+0000 7fa12cff9640 20 client.4120.objecter dump_active .. 0 homeless 2024-05-15T12:42:06.599 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.609+0000 7fa12cff9640 1 -- 172.21.15.178:0/778828130 <== mon.2 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fa12402d020 con 0x7fa1301d5e20 2024-05-15T12:42:06.599 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.609+0000 7fa12cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:42:06.599 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.609+0000 7fa12cff9640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:42:06.672 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.682+0000 7fa13626a640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "aaffd6d9-1a00-4793-9d1f-65a30501bec4", "id": 0}] 2024-05-15T12:42:06.672 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.682+0000 7fa13626a640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "aaffd6d9-1a00-4793-9d1f-65a30501bec4", "id": 0}] 2024-05-15T12:42:06.673 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.682+0000 7fa13626a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T12:42:06.673 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.682+0000 7fa13626a640 1 -- 172.21.15.178:0/778828130 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "osd new", "uuid": "aaffd6d9-1a00-4793-9d1f-65a30501bec4", "id": 0} v 0) v1 -- 0x7fa130067d90 con 0x7fa1301d5e20 2024-05-15T12:42:06.714 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-05-15T12:42:06.724+0000 7f5134c38040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-15T12:42:06.805 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-05-15T12:42:06.815+0000 7f5134c38040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-15T12:42:06.850 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-05-15T12:42:06.860+0000 7f5134c38040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-15T12:42:06.924 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.933+0000 7fa12cff9640 1 -- 172.21.15.178:0/778828130 <== mon.2 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "aaffd6d9-1a00-4793-9d1f-65a30501bec4", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7fa124021030 con 0x7fa1301d5e20 2024-05-15T12:42:06.924 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.933+0000 7fa12cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "aaffd6d9-1a00-4793-9d1f-65a30501bec4", "id": 0}] 2024-05-15T12:42:06.924 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.933+0000 7fa12cff9640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:42:06.924 INFO:teuthology.orchestra.run.smithi178.stdout:0 2024-05-15T12:42:06.926 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.934+0000 7fa13626a640 10 librados: watch_flush enter 2024-05-15T12:42:06.926 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.934+0000 7fa13626a640 10 librados: watch_flush exit 2024-05-15T12:42:06.926 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.934+0000 7fa13626a640 20 client.4120.objecter shutdown clearing up homeless session... 2024-05-15T12:42:06.926 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.934+0000 7fa13626a640 10 client.4120.objecter successfully canceled tick 2024-05-15T12:42:06.926 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.934+0000 7fa13626a640 10 monclient: shutdown 2024-05-15T12:42:06.926 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.934+0000 7fa13626a640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:42:06.927 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.934+0000 7fa13626a640 1 -- 172.21.15.178:0/778828130 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa1301d5e20 msgr2=0x7fa11c009df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:06.927 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.934+0000 7fa13626a640 1 --2- 172.21.15.178:0/778828130 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa1301d5e20 0x7fa11c009df0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa1300b4320 tx=0x7fa124007f10 comp rx=0 tx=0).stop 2024-05-15T12:42:06.927 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.934+0000 7fa13626a640 1 -- 172.21.15.178:0/778828130 shutdown_connections 2024-05-15T12:42:06.927 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.934+0000 7fa13626a640 1 --2- 172.21.15.178:0/778828130 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa130070bc0 0x7fa11c00c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:06.927 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.934+0000 7fa13626a640 1 --2- 172.21.15.178:0/778828130 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa1301d5e20 0x7fa11c009df0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:06.927 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.934+0000 7fa13626a640 1 --2- 172.21.15.178:0/778828130 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa11c004e30 0x7fa11c007280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:06.927 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.934+0000 7fa13626a640 1 -- 172.21.15.178:0/778828130 >> 172.21.15.178:0/778828130 conn(0x7fa1300b2ac0 msgr2=0x7fa13013a120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:42:06.927 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.935+0000 7fa13626a640 1 -- 172.21.15.178:0/778828130 shutdown_connections 2024-05-15T12:42:06.927 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.935+0000 7fa13626a640 1 -- 172.21.15.178:0/778828130 wait complete. 2024-05-15T12:42:06.927 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:06.935+0000 7fa13626a640 1 librados: shutdown 2024-05-15T12:42:06.933 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-05-15T12:42:06.943+0000 7f18296bb040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-15T12:42:06.935 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph --cluster ceph osd new efd4e8ed-699c-4ccb-b989-cfadc7aaed35 1 2024-05-15T12:42:07.032 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-05-15T12:42:07.042+0000 7f18296bb040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-15T12:42:07.037 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-05-15T12:42:07.047+0000 7f5134c38040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-15T12:42:07.040 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.050+0000 7f8d377b3640 10 monclient: get_monmap_and_config 2024-05-15T12:42:07.040 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.050+0000 7f8d377b3640 10 monclient: build_initial_monmap 2024-05-15T12:42:07.040 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.050+0000 7f8d377b3640 10 monclient: monmap: 2024-05-15T12:42:07.040 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 0 2024-05-15T12:42:07.040 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:07.040 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-05-15T12:42:07.051729+0000 2024-05-15T12:42:07.041 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-05-15T12:42:07.051729+0000 2024-05-15T12:42:07.041 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 0 (unknown) 2024-05-15T12:42:07.041 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2024-05-15T12:42:07.041 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T12:42:07.041 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T12:42:07.041 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T12:42:07.041 INFO:teuthology.orchestra.run.smithi178.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T12:42:07.041 INFO:teuthology.orchestra.run.smithi178.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T12:42:07.041 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-05-15T12:42:07.043 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.051+0000 7f8d377b3640 1 Processor -- start 2024-05-15T12:42:07.043 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.051+0000 7f8d377b3640 1 -- start start 2024-05-15T12:42:07.043 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.051+0000 7f8d377b3640 10 monclient: init 2024-05-15T12:42:07.043 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.052+0000 7f8d377b3640 10 monclient: _reopen_session rank -1 2024-05-15T12:42:07.043 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.052+0000 7f8d377b3640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-05-15T12:42:07.043 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.052+0000 7f8d377b3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8d300b4d00 addr v1:172.21.15.117:6789/0 2024-05-15T12:42:07.043 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.052+0000 7f8d377b3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8d300b50e0 addr v1:172.21.15.178:6789/0 2024-05-15T12:42:07.043 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.052+0000 7f8d377b3640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f8d300b1e80 0x7f8d300b0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:07.044 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.052+0000 7f8d377b3640 10 monclient(hunting): picked mon.noname-c con 0x7f8d300b1e80 addr v2:172.21.15.178:3300/0 2024-05-15T12:42:07.044 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.052+0000 7f8d377b3640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d300b2330 con 0x7f8d300b4d00 2024-05-15T12:42:07.044 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.052+0000 7f8d377b3640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d300ab1b0 con 0x7f8d300b50e0 2024-05-15T12:42:07.044 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.052+0000 7f8d377b3640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:07.044 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.052+0000 7f8d377b3640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d300b1520 con 0x7f8d300b1e80 2024-05-15T12:42:07.044 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.052+0000 7f8d377b3640 10 monclient(hunting): _renew_subs 2024-05-15T12:42:07.044 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.052+0000 7f8d377b3640 10 monclient(hunting): authenticate will time out at 1283.389620s 2024-05-15T12:42:07.044 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.052+0000 7f8d35d50640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f8d300b1e80 0x7f8d300b0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:07.044 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.053+0000 7f8d3554f640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f8d300b4d00 0x7f8d30145c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.178:49750/0 (socket says 172.21.15.178:49750) 2024-05-15T12:42:07.044 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.053+0000 7f8d3554f640 1 -- 172.21.15.178:0/2684025561 learned_addr learned my addr 172.21.15.178:0/2684025561 (peer_addr_for_me v1:172.21.15.178:0/0) 2024-05-15T12:42:07.045 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.053+0000 7f8d35d50640 10 monclient(hunting): get_auth_request con 0x7f8d300b1e80 auth_method 0 2024-05-15T12:42:07.045 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.053+0000 7f8d35d50640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:07.045 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.053+0000 7f8d35d50640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:07.045 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.053+0000 7f8d35d50640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:07.045 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.054+0000 7f8d35d50640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:07.045 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.054+0000 7f8d35d50640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:07.045 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.054+0000 7f8d35d50640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:07.046 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.056+0000 7f8d35d50640 10 monclient(hunting): handle_auth_done global_id 4123 payload 274 2024-05-15T12:42:07.046 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.056+0000 7f8d35d50640 1 -- 172.21.15.178:0/2684025561 >> v1:172.21.15.178:6789/0 conn(0x7f8d300b50e0 legacy=0x7f8d300a94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:07.046 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.056+0000 7f8d35d50640 1 -- 172.21.15.178:0/2684025561 >> v1:172.21.15.117:6789/0 conn(0x7f8d300b4d00 legacy=0x7f8d30145c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:07.047 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.056+0000 7f8d35d50640 10 monclient: _finish_hunting 0 2024-05-15T12:42:07.047 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.056+0000 7f8d35d50640 1 monclient: found mon.noname-c 2024-05-15T12:42:07.047 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.056+0000 7f8d35d50640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:42:07.047 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.056+0000 7f8d35d50640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.178:3300/0 2024-05-15T12:42:07.047 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.056+0000 7f8d35d50640 1 -- 172.21.15.178:0/2684025561 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d300b16a0 con 0x7f8d300b1e80 2024-05-15T12:42:07.051 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.061+0000 7f8d35d50640 1 --2- 172.21.15.178:0/2684025561 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8d300b1e80 0x7f8d300b0fe0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8d28009d90 tx=0x7f8d2802beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=73025789717a6876 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:42:07.057 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.066+0000 7f8d27fff640 1 -- 172.21.15.178:0/2684025561 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d28039070 con 0x7f8d300b1e80 2024-05-15T12:42:07.057 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.066+0000 7f8d27fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:07.057 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.066+0000 7f8d27fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:42:07.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.066+0000 7f8d27fff640 10 monclient: dump: 2024-05-15T12:42:07.058 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-05-15T12:42:07.058 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:07.058 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:07.058 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:07.058 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:07.058 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2024-05-15T12:42:07.058 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:07.058 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:07.058 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:07.058 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-05-15T12:42:07.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.066+0000 7f8d27fff640 10 monclient: _finish_auth 0 2024-05-15T12:42:07.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.066+0000 7f8d27fff640 10 monclient: _check_auth_tickets 2024-05-15T12:42:07.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.066+0000 7f8d27fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:42:07.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.066+0000 7f8d27fff640 1 -- 172.21.15.178:0/2684025561 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d2802cc60 con 0x7f8d300b1e80 2024-05-15T12:42:07.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.066+0000 7f8d27fff640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:42:07.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.066+0000 7f8d377b3640 5 monclient: authenticate success, global_id 4123 2024-05-15T12:42:07.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.066+0000 7f8d27fff640 1 -- 172.21.15.178:0/2684025561 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d28034660 con 0x7f8d300b1e80 2024-05-15T12:42:07.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.066+0000 7f8d377b3640 10 monclient: get_monmap_and_config success 2024-05-15T12:42:07.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.066+0000 7f8d27fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:07.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.066+0000 7f8d377b3640 4 set_mon_vals no callback set 2024-05-15T12:42:07.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.066+0000 7f8d27fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:42:07.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.066+0000 7f8d27fff640 10 monclient: dump: 2024-05-15T12:42:07.059 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-05-15T12:42:07.059 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:07.060 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:07.060 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:07.060 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:07.060 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2024-05-15T12:42:07.060 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:07.060 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:07.060 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:07.060 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-05-15T12:42:07.060 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.067+0000 7f8d377b3640 10 monclient: shutdown 2024-05-15T12:42:07.060 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.067+0000 7f8d377b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:42:07.060 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.067+0000 7f8d377b3640 1 -- 172.21.15.178:0/2684025561 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8d300b1e80 msgr2=0x7f8d300b0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:07.060 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.067+0000 7f8d377b3640 1 --2- 172.21.15.178:0/2684025561 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8d300b1e80 0x7f8d300b0fe0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8d28009d90 tx=0x7f8d2802beb0 comp rx=0 tx=0).stop 2024-05-15T12:42:07.060 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.067+0000 7f8d377b3640 1 -- 172.21.15.178:0/2684025561 shutdown_connections 2024-05-15T12:42:07.061 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.067+0000 7f8d377b3640 1 --2- 172.21.15.178:0/2684025561 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8d300b1e80 0x7f8d300b0fe0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:07.061 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.067+0000 7f8d377b3640 1 -- 172.21.15.178:0/2684025561 >> 172.21.15.178:0/2684025561 conn(0x7f8d30066cc0 msgr2=0x7f8d300670d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:42:07.061 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.067+0000 7f8d377b3640 1 -- 172.21.15.178:0/2684025561 shutdown_connections 2024-05-15T12:42:07.061 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.067+0000 7f8d377b3640 1 -- 172.21.15.178:0/2684025561 wait complete. 2024-05-15T12:42:07.064 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.068+0000 7f8d377b3640 10 monclient: build_initial_monmap 2024-05-15T12:42:07.064 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.068+0000 7f8d377b3640 10 monclient: monmap: 2024-05-15T12:42:07.064 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 0 2024-05-15T12:42:07.064 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:42:07.064 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 0.000000 2024-05-15T12:42:07.064 INFO:teuthology.orchestra.run.smithi178.stderr:created 0.000000 2024-05-15T12:42:07.065 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 0 (unknown) 2024-05-15T12:42:07.065 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2024-05-15T12:42:07.065 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T12:42:07.065 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T12:42:07.065 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T12:42:07.065 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-05-15T12:42:07.065 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.068+0000 7f8d377b3640 1 librados: starting msgr at 2024-05-15T12:42:07.065 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.068+0000 7f8d377b3640 1 librados: starting objecter 2024-05-15T12:42:07.065 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.068+0000 7f8d377b3640 1 Processor -- start 2024-05-15T12:42:07.065 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.069+0000 7f8d377b3640 1 -- start start 2024-05-15T12:42:07.065 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.069+0000 7f8d377b3640 1 librados: setting wanted keys 2024-05-15T12:42:07.065 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.069+0000 7f8d377b3640 1 librados: calling monclient init 2024-05-15T12:42:07.065 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.069+0000 7f8d377b3640 10 monclient: init 2024-05-15T12:42:07.065 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.069+0000 7f8d377b3640 10 monclient: _reopen_session rank -1 2024-05-15T12:42:07.065 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.069+0000 7f8d377b3640 10 monclient: _add_conns ranks=[0,2,1] 2024-05-15T12:42:07.065 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.069+0000 7f8d377b3640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d300b1e80 0x7f8d3013d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:07.066 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.069+0000 7f8d377b3640 10 monclient(hunting): picked mon.noname-a con 0x7f8d300b1e80 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T12:42:07.066 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.069+0000 7f8d377b3640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8d300b4d00 0x7f8d300ac790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:07.066 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.069+0000 7f8d377b3640 10 monclient(hunting): picked mon.noname-c con 0x7f8d300b4d00 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:42:07.066 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.069+0000 7f8d377b3640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8d300ae4e0 0x7f8d3013c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:07.066 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.069+0000 7f8d377b3640 10 monclient(hunting): picked mon.noname-b con 0x7f8d300ae4e0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:42:07.066 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.069+0000 7f8d377b3640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:07.066 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.069+0000 7f8d377b3640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3014e350 con 0x7f8d300b1e80 2024-05-15T12:42:07.066 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.069+0000 7f8d377b3640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:07.066 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.069+0000 7f8d377b3640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3014e1d0 con 0x7f8d300ae4e0 2024-05-15T12:42:07.066 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.069+0000 7f8d377b3640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:07.066 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.069+0000 7f8d377b3640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3014e4d0 con 0x7f8d300b4d00 2024-05-15T12:42:07.066 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.069+0000 7f8d377b3640 10 monclient(hunting): _renew_subs 2024-05-15T12:42:07.066 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.069+0000 7f8d377b3640 10 monclient(hunting): authenticate will time out at 1283.406953s 2024-05-15T12:42:07.066 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.070+0000 7f8d35d50640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8d300ae4e0 0x7f8d3013c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:07.067 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.070+0000 7f8d3554f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d300b1e80 0x7f8d3013d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:07.067 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.070+0000 7f8d34d4e640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8d300b4d00 0x7f8d300ac790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:07.067 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.070+0000 7f8d35d50640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8d300ae4e0 0x7f8d3013c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.178:44906/0 (socket says 172.21.15.178:44906) 2024-05-15T12:42:07.067 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.070+0000 7f8d35d50640 1 -- 172.21.15.178:0/1046793629 learned_addr learned my addr 172.21.15.178:0/1046793629 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-05-15T12:42:07.067 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.070+0000 7f8d35d50640 10 monclient(hunting): get_auth_request con 0x7f8d300ae4e0 auth_method 0 2024-05-15T12:42:07.067 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.070+0000 7f8d35d50640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:07.067 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.070+0000 7f8d35d50640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:07.067 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.070+0000 7f8d35d50640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:07.067 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.070+0000 7f8d34d4e640 10 monclient(hunting): get_auth_request con 0x7f8d300b4d00 auth_method 0 2024-05-15T12:42:07.067 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.070+0000 7f8d34d4e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:07.067 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.070+0000 7f8d34d4e640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:07.067 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.070+0000 7f8d34d4e640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:07.067 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.070+0000 7f8d3554f640 10 monclient(hunting): get_auth_request con 0x7f8d300b1e80 auth_method 0 2024-05-15T12:42:07.067 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.070+0000 7f8d3554f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:07.068 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.070+0000 7f8d3554f640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:07.068 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.070+0000 7f8d3554f640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:07.068 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.071+0000 7f8d35d50640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:07.068 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.071+0000 7f8d35d50640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:07.068 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.071+0000 7f8d35d50640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:07.068 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.071+0000 7f8d34d4e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:07.068 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.071+0000 7f8d34d4e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:07.068 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.071+0000 7f8d34d4e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:07.068 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.071+0000 7f8d3554f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:07.068 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.071+0000 7f8d3554f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:07.068 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.071+0000 7f8d3554f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:07.068 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.073+0000 7f8d35d50640 10 monclient(hunting): handle_auth_done global_id 4129 payload 824 2024-05-15T12:42:07.068 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.073+0000 7f8d35d50640 1 -- 172.21.15.178:0/1046793629 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8d300b4d00 msgr2=0x7f8d300ac790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:07.068 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.073+0000 7f8d35d50640 1 --2- 172.21.15.178:0/1046793629 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8d300b4d00 0x7f8d300ac790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:07.068 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.073+0000 7f8d35d50640 1 -- 172.21.15.178:0/1046793629 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d300b1e80 msgr2=0x7f8d3013d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:07.069 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.073+0000 7f8d35d50640 1 --2- 172.21.15.178:0/1046793629 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d300b1e80 0x7f8d3013d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:07.069 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.073+0000 7f8d35d50640 10 monclient: _finish_hunting 0 2024-05-15T12:42:07.069 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.073+0000 7f8d35d50640 1 monclient: found mon.noname-b 2024-05-15T12:42:07.069 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.073+0000 7f8d35d50640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:42:07.069 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.073+0000 7f8d35d50640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2024-05-15T12:42:07.069 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.073+0000 7f8d35d50640 1 -- 172.21.15.178:0/1046793629 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d300adb70 con 0x7f8d300ae4e0 2024-05-15T12:42:07.069 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.077+0000 7f8d35d50640 1 -- 172.21.15.178:0/1046793629 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8d300ae4e0 msgr2=0x7f8d3013c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-05-15T12:42:07.069 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.077+0000 7f8d35d50640 1 -- 172.21.15.178:0/1046793629 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8d300ae4e0 msgr2=0x7f8d3013c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-15T12:42:07.069 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.077+0000 7f8d35d50640 1 --2- 172.21.15.178:0/1046793629 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8d300ae4e0 0x7f8d3013c620 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f8d2802c4c0 tx=0x7f8d28004a40 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-15T12:42:07.070 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.077+0000 7f8d35d50640 1 --2- 172.21.15.178:0/1046793629 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8d300ae4e0 0x7f8d3013c620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-15T12:42:07.080 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-05-15T12:42:07.091+0000 7f18296bb040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-15T12:42:07.099 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-05-15T12:42:07.109+0000 7f5134c38040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-15T12:42:07.153 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-05-15T12:42:07.163+0000 7f5134c38040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-15T12:42:07.245 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-05-15T12:42:07.255+0000 7f18296bb040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-15T12:42:07.262 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-05-15T12:42:07.272+0000 7f5134c38040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-15T12:42:07.268 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.278+0000 7f8d35d50640 1 --2- 172.21.15.178:0/1046793629 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8d300ae4e0 0x7f8d3013c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:07.271 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.279+0000 7f8d35d50640 10 monclient: get_auth_request con 0x7f8d300ae4e0 auth_method 0 2024-05-15T12:42:07.271 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.279+0000 7f8d35d50640 0 --2- 172.21.15.178:0/1046793629 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8d300ae4e0 0x7f8d3013c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-05-15T12:42:07.271 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.279+0000 7f8d35d50640 1 --2- 172.21.15.178:0/1046793629 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8d300ae4e0 0x7f8d3013c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:07.271 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.279+0000 7f8d25ffb640 10 monclient: ms_handle_reset current mon [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:42:07.271 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.279+0000 7f8d25ffb640 10 monclient: _reopen_session rank -1 2024-05-15T12:42:07.271 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.279+0000 7f8d25ffb640 1 -- 172.21.15.178:0/1046793629 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8d300ae4e0 msgr2=0x7f8d3013c620 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-15T12:42:07.271 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.279+0000 7f8d25ffb640 1 --2- 172.21.15.178:0/1046793629 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8d300ae4e0 0x7f8d3013c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:07.271 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.279+0000 7f8d25ffb640 10 monclient: _add_conns ranks=[0,2,1] 2024-05-15T12:42:07.271 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.279+0000 7f8d25ffb640 1 --2- 172.21.15.178:0/1046793629 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d300b1e80 0x7f8d140054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:07.271 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.279+0000 7f8d25ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f8d300b1e80 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T12:42:07.271 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.279+0000 7f8d25ffb640 1 --2- 172.21.15.178:0/1046793629 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8d300b4d00 0x7f8d14007fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:07.272 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.279+0000 7f8d25ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f8d300b4d00 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:42:07.272 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.279+0000 7f8d25ffb640 1 --2- 172.21.15.178:0/1046793629 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8d14008a00 0x7f8d1400ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:07.272 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.279+0000 7f8d25ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f8d14008a00 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:42:07.272 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.279+0000 7f8d25ffb640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:07.272 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.279+0000 7f8d25ffb640 1 -- 172.21.15.178:0/1046793629 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d1400b990 con 0x7f8d300b1e80 2024-05-15T12:42:07.272 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.279+0000 7f8d25ffb640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:07.272 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.279+0000 7f8d25ffb640 1 -- 172.21.15.178:0/1046793629 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d1400bb10 con 0x7f8d14008a00 2024-05-15T12:42:07.272 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.279+0000 7f8d25ffb640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:07.272 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.279+0000 7f8d25ffb640 1 -- 172.21.15.178:0/1046793629 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d1400bd10 con 0x7f8d300b4d00 2024-05-15T12:42:07.272 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.279+0000 7f8d25ffb640 10 monclient(hunting): _renew_subs 2024-05-15T12:42:07.272 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.279+0000 7f8d3554f640 1 --2- 172.21.15.178:0/1046793629 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d300b1e80 0x7f8d140054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:07.272 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.280+0000 7f8d34d4e640 1 --2- 172.21.15.178:0/1046793629 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8d300b4d00 0x7f8d14007fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:07.272 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.280+0000 7f8d3554f640 10 monclient(hunting): get_auth_request con 0x7f8d300b1e80 auth_method 0 2024-05-15T12:42:07.272 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.280+0000 7f8d3554f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:07.272 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.280+0000 7f8d3554f640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:07.272 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.280+0000 7f8d3554f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-05-15T12:42:07.273 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.280+0000 7f8d34d4e640 10 monclient(hunting): get_auth_request con 0x7f8d300b4d00 auth_method 0 2024-05-15T12:42:07.273 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.280+0000 7f8d34d4e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:07.273 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.280+0000 7f8d34d4e640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:07.273 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.280+0000 7f8d34d4e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-05-15T12:42:07.273 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.280+0000 7f8d35d50640 1 --2- 172.21.15.178:0/1046793629 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8d14008a00 0x7f8d1400ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:07.273 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.280+0000 7f8d3554f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:07.273 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.280+0000 7f8d3554f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:07.273 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.280+0000 7f8d3554f640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-05-15T12:42:07.273 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.281+0000 7f8d35d50640 10 monclient(hunting): get_auth_request con 0x7f8d14008a00 auth_method 0 2024-05-15T12:42:07.273 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.281+0000 7f8d35d50640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:07.273 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.281+0000 7f8d35d50640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:07.273 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.281+0000 7f8d35d50640 10 monclient(hunting): _init_auth already have auth, reseting 2024-05-15T12:42:07.273 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.281+0000 7f8d34d4e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:07.273 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.281+0000 7f8d34d4e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:07.273 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.281+0000 7f8d34d4e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-05-15T12:42:07.274 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.281+0000 7f8d35d50640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:07.274 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.281+0000 7f8d35d50640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:07.274 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.281+0000 7f8d35d50640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-05-15T12:42:07.275 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.282+0000 7f8d3554f640 10 monclient(hunting): handle_auth_done global_id 4129 payload 293 2024-05-15T12:42:07.275 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.282+0000 7f8d3554f640 1 -- 172.21.15.178:0/1046793629 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8d300b4d00 msgr2=0x7f8d14007fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:07.275 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.282+0000 7f8d3554f640 1 --2- 172.21.15.178:0/1046793629 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8d300b4d00 0x7f8d14007fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:07.275 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.282+0000 7f8d3554f640 1 -- 172.21.15.178:0/1046793629 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8d14008a00 msgr2=0x7f8d1400ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:07.275 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.282+0000 7f8d3554f640 1 --2- 172.21.15.178:0/1046793629 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8d14008a00 0x7f8d1400ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:07.275 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.282+0000 7f8d3554f640 10 monclient: _finish_hunting 0 2024-05-15T12:42:07.275 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.282+0000 7f8d3554f640 1 monclient: found mon.noname-a 2024-05-15T12:42:07.275 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.282+0000 7f8d3554f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:42:07.275 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.282+0000 7f8d3554f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.117:3300/0 2024-05-15T12:42:07.275 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.282+0000 7f8d3554f640 1 -- 172.21.15.178:0/1046793629 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d1400bfd0 con 0x7f8d300b1e80 2024-05-15T12:42:07.275 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.284+0000 7f8d3554f640 1 --2- 172.21.15.178:0/1046793629 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d300b1e80 0x7f8d140054d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c000f60 tx=0x7f8d28030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:42:07.275 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.284+0000 7f8d25ffb640 10 client.?.objecter ms_handle_connect 0x7f8d300b1e80 2024-05-15T12:42:07.275 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.284+0000 7f8d25ffb640 10 client.?.objecter resend_mon_ops 2024-05-15T12:42:07.279 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.289+0000 7f8d25ffb640 1 -- 172.21.15.178:0/1046793629 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d2c001590 con 0x7f8d300b1e80 2024-05-15T12:42:07.279 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.289+0000 7f8d25ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:07.279 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.289+0000 7f8d25ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T12:42:07.279 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.289+0000 7f8d25ffb640 10 monclient: dump: 2024-05-15T12:42:07.279 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-05-15T12:42:07.279 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:07.279 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:07.279 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:07.280 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:07.280 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2024-05-15T12:42:07.280 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:07.280 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:07.280 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:07.280 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-05-15T12:42:07.280 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.289+0000 7f8d25ffb640 10 monclient: _finish_auth 0 2024-05-15T12:42:07.280 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.289+0000 7f8d25ffb640 10 monclient: _check_auth_tickets 2024-05-15T12:42:07.280 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.289+0000 7f8d25ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:42:07.280 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.289+0000 7f8d25ffb640 1 -- 172.21.15.178:0/1046793629 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d2c018070 con 0x7f8d300b1e80 2024-05-15T12:42:07.280 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.289+0000 7f8d25ffb640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:42:07.280 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.289+0000 7f8d25ffb640 1 -- 172.21.15.178:0/1046793629 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d2c005af0 con 0x7f8d300b1e80 2024-05-15T12:42:07.280 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.289+0000 7f8d25ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:07.280 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.289+0000 7f8d25ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T12:42:07.280 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.289+0000 7f8d25ffb640 10 monclient: dump: 2024-05-15T12:42:07.280 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-05-15T12:42:07.281 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:07.281 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:07.281 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:07.281 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:07.281 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2024-05-15T12:42:07.281 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:07.281 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:07.281 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:07.281 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-05-15T12:42:07.281 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.289+0000 7f8d377b3640 5 monclient: authenticate success, global_id 4129 2024-05-15T12:42:07.281 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.289+0000 7f8d377b3640 10 monclient: _renew_subs 2024-05-15T12:42:07.281 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.289+0000 7f8d377b3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:42:07.281 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.289+0000 7f8d377b3640 1 -- 172.21.15.178:0/1046793629 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d300aba60 con 0x7f8d300b1e80 2024-05-15T12:42:07.281 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.289+0000 7f8d277fe640 4 set_mon_vals no callback set 2024-05-15T12:42:07.281 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.289+0000 7f8d377b3640 10 client.4129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:42:07.282 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.289+0000 7f8d377b3640 10 monclient: _renew_subs 2024-05-15T12:42:07.282 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.289+0000 7f8d377b3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:42:07.282 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.289+0000 7f8d377b3640 1 -- 172.21.15.178:0/1046793629 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d300ae0c0 con 0x7f8d300b1e80 2024-05-15T12:42:07.282 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.289+0000 7f8d377b3640 1 librados: init done 2024-05-15T12:42:07.283 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.290+0000 7f8d377b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:42:07.283 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.290+0000 7f8d377b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:42:07.283 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.290+0000 7f8d377b3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:42:07.283 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.290+0000 7f8d377b3640 1 -- 172.21.15.178:0/1046793629 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d3004f0e0 con 0x7f8d300b1e80 2024-05-15T12:42:07.283 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.292+0000 7f8d25ffb640 1 -- 172.21.15.178:0/1046793629 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f8d2c005c90 con 0x7f8d300b1e80 2024-05-15T12:42:07.283 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.292+0000 7f8d25ffb640 1 -- 172.21.15.178:0/1046793629 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f8d2c01d460 con 0x7f8d300b1e80 2024-05-15T12:42:07.283 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.292+0000 7f8d25ffb640 10 client.4129.objecter ms_dispatch 0x7f8d30000b90 osd_map(3..3 src has 1..3) v4 2024-05-15T12:42:07.283 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.292+0000 7f8d25ffb640 3 client.4129.objecter handle_osd_map got epochs [3,3] > 0 2024-05-15T12:42:07.283 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.292+0000 7f8d25ffb640 3 client.4129.objecter handle_osd_map decoding full epoch 3 2024-05-15T12:42:07.283 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.292+0000 7f8d25ffb640 20 client.4129.objecter dump_active .. 0 homeless 2024-05-15T12:42:07.301 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-05-15T12:42:07.311+0000 7f18296bb040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-15T12:42:07.352 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-05-15T12:42:07.362+0000 7f18296bb040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-15T12:42:07.400 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.409+0000 7f8d25ffb640 1 -- 172.21.15.178:0/1046793629 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f8d2c0013b0 con 0x7f8d300b1e80 2024-05-15T12:42:07.400 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.409+0000 7f8d25ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:42:07.400 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.409+0000 7f8d25ffb640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:42:07.464 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-05-15T12:42:07.474+0000 7f18296bb040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-15T12:42:07.469 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.479+0000 7f8d377b3640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "efd4e8ed-699c-4ccb-b989-cfadc7aaed35", "id": 1}] 2024-05-15T12:42:07.469 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.479+0000 7f8d377b3640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "efd4e8ed-699c-4ccb-b989-cfadc7aaed35", "id": 1}] 2024-05-15T12:42:07.469 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.479+0000 7f8d377b3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:42:07.470 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.479+0000 7f8d377b3640 1 -- 172.21.15.178:0/1046793629 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd new", "uuid": "efd4e8ed-699c-4ccb-b989-cfadc7aaed35", "id": 1} v 0) v1 -- 0x7f8d300b7160 con 0x7f8d300b1e80 2024-05-15T12:42:07.512 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.522+0000 7f8d25ffb640 1 -- 172.21.15.178:0/1046793629 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "efd4e8ed-699c-4ccb-b989-cfadc7aaed35", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f8d2c00f8d0 con 0x7f8d300b1e80 2024-05-15T12:42:07.512 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.522+0000 7f8d25ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "efd4e8ed-699c-4ccb-b989-cfadc7aaed35", "id": 1}] 2024-05-15T12:42:07.512 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.522+0000 7f8d25ffb640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:42:07.512 INFO:teuthology.orchestra.run.smithi178.stdout:1 2024-05-15T12:42:07.514 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.523+0000 7f8d377b3640 10 librados: watch_flush enter 2024-05-15T12:42:07.514 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.523+0000 7f8d377b3640 10 librados: watch_flush exit 2024-05-15T12:42:07.514 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.523+0000 7f8d377b3640 20 client.4129.objecter shutdown clearing up homeless session... 2024-05-15T12:42:07.514 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.523+0000 7f8d377b3640 10 client.4129.objecter successfully canceled tick 2024-05-15T12:42:07.514 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.523+0000 7f8d377b3640 10 monclient: shutdown 2024-05-15T12:42:07.514 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.523+0000 7f8d377b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:42:07.514 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.523+0000 7f8d377b3640 1 -- 172.21.15.178:0/1046793629 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d300b1e80 msgr2=0x7f8d140054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:07.514 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.523+0000 7f8d377b3640 1 --2- 172.21.15.178:0/1046793629 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d300b1e80 0x7f8d140054d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c000f60 tx=0x7f8d28030000 comp rx=0 tx=0).stop 2024-05-15T12:42:07.514 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.523+0000 7f8d377b3640 1 -- 172.21.15.178:0/1046793629 shutdown_connections 2024-05-15T12:42:07.514 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.523+0000 7f8d377b3640 1 --2- 172.21.15.178:0/1046793629 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8d14008a00 0x7f8d1400ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:07.514 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.523+0000 7f8d377b3640 1 --2- 172.21.15.178:0/1046793629 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8d300b4d00 0x7f8d14007fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:07.514 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.523+0000 7f8d377b3640 1 --2- 172.21.15.178:0/1046793629 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d300b1e80 0x7f8d140054d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:07.515 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.523+0000 7f8d377b3640 1 -- 172.21.15.178:0/1046793629 >> 172.21.15.178:0/1046793629 conn(0x7f8d30066cc0 msgr2=0x7f8d3013f6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:42:07.515 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.523+0000 7f8d377b3640 1 -- 172.21.15.178:0/1046793629 shutdown_connections 2024-05-15T12:42:07.515 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.523+0000 7f8d377b3640 1 -- 172.21.15.178:0/1046793629 wait complete. 2024-05-15T12:42:07.515 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.523+0000 7f8d377b3640 1 librados: shutdown 2024-05-15T12:42:07.524 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph --cluster ceph osd new 708e4e3e-7e79-480f-9dbf-7b972c2f2811 2 2024-05-15T12:42:07.622 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.631+0000 7f0772356640 10 monclient: get_monmap_and_config 2024-05-15T12:42:07.622 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.631+0000 7f0772356640 10 monclient: build_initial_monmap 2024-05-15T12:42:07.622 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.631+0000 7f0772356640 10 monclient: monmap: 2024-05-15T12:42:07.622 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 0 2024-05-15T12:42:07.622 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:07.622 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-05-15T12:42:07.632870+0000 2024-05-15T12:42:07.622 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-05-15T12:42:07.632870+0000 2024-05-15T12:42:07.622 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 0 (unknown) 2024-05-15T12:42:07.622 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2024-05-15T12:42:07.622 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T12:42:07.622 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T12:42:07.623 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T12:42:07.623 INFO:teuthology.orchestra.run.smithi178.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T12:42:07.623 INFO:teuthology.orchestra.run.smithi178.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T12:42:07.623 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-05-15T12:42:07.624 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.632+0000 7f0772356640 1 Processor -- start 2024-05-15T12:42:07.624 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.632+0000 7f0772356640 1 -- start start 2024-05-15T12:42:07.625 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.632+0000 7f0772356640 10 monclient: init 2024-05-15T12:42:07.625 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.633+0000 7f0772356640 10 monclient: _reopen_session rank -1 2024-05-15T12:42:07.625 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.633+0000 7f0772356640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-05-15T12:42:07.625 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.633+0000 7f0772356640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f076c0b4d00 addr v1:172.21.15.117:6789/0 2024-05-15T12:42:07.625 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.633+0000 7f0772356640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f076c0b50e0 addr v1:172.21.15.178:6789/0 2024-05-15T12:42:07.625 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.633+0000 7f0772356640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f076c0b1e80 0x7f076c0b0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:07.625 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.633+0000 7f0772356640 10 monclient(hunting): picked mon.noname-a con 0x7f076c0b1e80 addr v2:172.21.15.117:3300/0 2024-05-15T12:42:07.625 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.633+0000 7f0772356640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f076c0b2330 con 0x7f076c0b4d00 2024-05-15T12:42:07.625 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.633+0000 7f0772356640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f076c0ab1b0 con 0x7f076c0b50e0 2024-05-15T12:42:07.625 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.633+0000 7f0772356640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:07.625 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.633+0000 7f0772356640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f076c0b1520 con 0x7f076c0b1e80 2024-05-15T12:42:07.625 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.633+0000 7f0772356640 10 monclient(hunting): _renew_subs 2024-05-15T12:42:07.625 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.633+0000 7f0772356640 10 monclient(hunting): authenticate will time out at 1283.970847s 2024-05-15T12:42:07.625 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.634+0000 7f0763fff640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f076c0b50e0 0x7f076c0a94b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.178:60856/0 (socket says 172.21.15.178:60856) 2024-05-15T12:42:07.626 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.634+0000 7f0763fff640 1 -- 172.21.15.178:0/1279837913 learned_addr learned my addr 172.21.15.178:0/1279837913 (peer_addr_for_me v1:172.21.15.178:0/0) 2024-05-15T12:42:07.626 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.634+0000 7f07708f3640 1 --2- 172.21.15.178:0/1279837913 >> v2:172.21.15.117:3300/0 conn(0x7f076c0b1e80 0x7f076c0b0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:07.626 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.634+0000 7f07708f3640 10 monclient(hunting): get_auth_request con 0x7f076c0b1e80 auth_method 0 2024-05-15T12:42:07.626 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.634+0000 7f07708f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:07.626 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.634+0000 7f07708f3640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:07.626 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.634+0000 7f07708f3640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:07.627 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.635+0000 7f07708f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:07.627 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.635+0000 7f07708f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:07.627 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.635+0000 7f07708f3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:07.627 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.636+0000 7f076b7fe640 1 -- 172.21.15.178:0/1279837913 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2575676979 0 0) 0x7f076c0ab1b0 con 0x7f076c0b50e0 2024-05-15T12:42:07.627 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.636+0000 7f076b7fe640 10 monclient(hunting): _negotiate 2024-05-15T12:42:07.627 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.636+0000 7f076b7fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:07.627 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.636+0000 7f076b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:07.627 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.636+0000 7f076b7fe640 10 monclient(hunting): my global_id is 4132 2024-05-15T12:42:07.627 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.636+0000 7f076b7fe640 1 -- 172.21.15.178:0/1279837913 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0758003520 con 0x7f076c0b50e0 2024-05-15T12:42:07.628 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.638+0000 7f07708f3640 10 monclient(hunting): handle_auth_done global_id 4122 payload 274 2024-05-15T12:42:07.628 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.638+0000 7f07708f3640 1 -- 172.21.15.178:0/1279837913 >> v1:172.21.15.178:6789/0 conn(0x7f076c0b50e0 legacy=0x7f076c0a94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:07.628 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.638+0000 7f07708f3640 1 -- 172.21.15.178:0/1279837913 >> v1:172.21.15.117:6789/0 conn(0x7f076c0b4d00 legacy=0x7f076c145c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:07.628 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.638+0000 7f07708f3640 10 monclient: _finish_hunting 0 2024-05-15T12:42:07.628 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.638+0000 7f07708f3640 1 monclient: found mon.noname-a 2024-05-15T12:42:07.628 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.638+0000 7f07708f3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:42:07.628 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.638+0000 7f07708f3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.117:3300/0 2024-05-15T12:42:07.628 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.638+0000 7f07708f3640 1 -- 172.21.15.178:0/1279837913 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f076c0b16a0 con 0x7f076c0b1e80 2024-05-15T12:42:07.631 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.642+0000 7f07708f3640 1 --2- 172.21.15.178:0/1279837913 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f076c0b1e80 0x7f076c0b0fe0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0764009d90 tx=0x7f076402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22979a9913efa202 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:42:07.634 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.644+0000 7f076b7fe640 1 -- 172.21.15.178:0/1279837913 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0764039070 con 0x7f076c0b1e80 2024-05-15T12:42:07.635 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.644+0000 7f076b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:07.635 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.644+0000 7f076b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T12:42:07.635 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.644+0000 7f076b7fe640 10 monclient: dump: 2024-05-15T12:42:07.635 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-05-15T12:42:07.635 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:07.635 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:07.635 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:07.635 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:07.635 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2024-05-15T12:42:07.635 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:07.635 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:07.635 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:07.635 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-05-15T12:42:07.635 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.644+0000 7f076b7fe640 10 monclient: _finish_auth 0 2024-05-15T12:42:07.635 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.644+0000 7f076b7fe640 10 monclient: _check_auth_tickets 2024-05-15T12:42:07.636 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.644+0000 7f076b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:42:07.636 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.644+0000 7f0772356640 5 monclient: authenticate success, global_id 4122 2024-05-15T12:42:07.636 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.644+0000 7f0772356640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:42:07.636 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.646+0000 7f076b7fe640 1 -- 172.21.15.178:0/1279837913 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f076402cc60 con 0x7f076c0b1e80 2024-05-15T12:42:07.636 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.646+0000 7f076b7fe640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:42:07.636 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.646+0000 7f076b7fe640 1 -- 172.21.15.178:0/1279837913 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0764034640 con 0x7f076c0b1e80 2024-05-15T12:42:07.636 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.646+0000 7f076b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:07.636 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.646+0000 7f076b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T12:42:07.636 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.646+0000 7f076b7fe640 10 monclient: dump: 2024-05-15T12:42:07.637 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-05-15T12:42:07.637 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:07.637 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:07.637 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:07.637 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:07.637 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2024-05-15T12:42:07.637 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:07.637 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:07.637 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:07.637 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-05-15T12:42:07.637 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.646+0000 7f0772356640 10 monclient: get_monmap_and_config success 2024-05-15T12:42:07.637 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.646+0000 7f0772356640 4 set_mon_vals no callback set 2024-05-15T12:42:07.641 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.647+0000 7f0772356640 10 monclient: shutdown 2024-05-15T12:42:07.641 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.647+0000 7f0772356640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:42:07.641 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.647+0000 7f0772356640 1 -- 172.21.15.178:0/1279837913 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f076c0b1e80 msgr2=0x7f076c0b0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:07.641 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.647+0000 7f0772356640 1 --2- 172.21.15.178:0/1279837913 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f076c0b1e80 0x7f076c0b0fe0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0764009d90 tx=0x7f076402beb0 comp rx=0 tx=0).stop 2024-05-15T12:42:07.642 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.647+0000 7f0772356640 1 -- 172.21.15.178:0/1279837913 shutdown_connections 2024-05-15T12:42:07.642 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.647+0000 7f0772356640 1 --2- 172.21.15.178:0/1279837913 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f076c0b1e80 0x7f076c0b0fe0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:07.642 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.647+0000 7f0772356640 1 -- 172.21.15.178:0/1279837913 >> 172.21.15.178:0/1279837913 conn(0x7f076c066cc0 msgr2=0x7f076c0670d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:42:07.642 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.647+0000 7f0772356640 1 -- 172.21.15.178:0/1279837913 shutdown_connections 2024-05-15T12:42:07.642 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.647+0000 7f0772356640 1 -- 172.21.15.178:0/1279837913 wait complete. 2024-05-15T12:42:07.642 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.648+0000 7f0772356640 10 monclient: build_initial_monmap 2024-05-15T12:42:07.642 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.648+0000 7f0772356640 10 monclient: monmap: 2024-05-15T12:42:07.642 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 0 2024-05-15T12:42:07.642 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:42:07.642 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 0.000000 2024-05-15T12:42:07.642 INFO:teuthology.orchestra.run.smithi178.stderr:created 0.000000 2024-05-15T12:42:07.642 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 0 (unknown) 2024-05-15T12:42:07.642 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2024-05-15T12:42:07.642 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T12:42:07.643 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T12:42:07.643 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T12:42:07.643 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-05-15T12:42:07.643 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.648+0000 7f0772356640 1 librados: starting msgr at 2024-05-15T12:42:07.643 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.648+0000 7f0772356640 1 librados: starting objecter 2024-05-15T12:42:07.643 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.648+0000 7f0772356640 1 Processor -- start 2024-05-15T12:42:07.643 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.648+0000 7f0772356640 1 -- start start 2024-05-15T12:42:07.643 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.648+0000 7f0772356640 1 librados: setting wanted keys 2024-05-15T12:42:07.643 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.648+0000 7f0772356640 1 librados: calling monclient init 2024-05-15T12:42:07.643 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.648+0000 7f0772356640 10 monclient: init 2024-05-15T12:42:07.643 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.648+0000 7f0772356640 10 monclient: _reopen_session rank -1 2024-05-15T12:42:07.643 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.648+0000 7f0772356640 10 monclient: _add_conns ranks=[0,2,1] 2024-05-15T12:42:07.643 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.648+0000 7f0772356640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f076c0b1e80 0x7f076c144b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:07.643 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.649+0000 7f0772356640 10 monclient(hunting): picked mon.noname-a con 0x7f076c0b1e80 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T12:42:07.643 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.649+0000 7f0772356640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f076c0b4d00 0x7f076c1410c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:07.644 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.649+0000 7f0772356640 10 monclient(hunting): picked mon.noname-c con 0x7f076c0b4d00 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:42:07.644 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.649+0000 7f0772356640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f076c141600 0x7f076c137830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:07.644 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.649+0000 7f0772356640 10 monclient(hunting): picked mon.noname-b con 0x7f076c141600 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:42:07.644 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.649+0000 7f0772356640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:07.644 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.649+0000 7f0772356640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f076c14e350 con 0x7f076c0b1e80 2024-05-15T12:42:07.644 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.649+0000 7f0772356640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:07.644 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.649+0000 7f0772356640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f076c14e1d0 con 0x7f076c141600 2024-05-15T12:42:07.644 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.649+0000 7f0772356640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:07.644 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.649+0000 7f0772356640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f076c14e4d0 con 0x7f076c0b4d00 2024-05-15T12:42:07.644 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.649+0000 7f0772356640 10 monclient(hunting): _renew_subs 2024-05-15T12:42:07.644 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.649+0000 7f0772356640 10 monclient(hunting): authenticate will time out at 1283.986437s 2024-05-15T12:42:07.644 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.649+0000 7f076bfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f076c0b1e80 0x7f076c144b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:07.644 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.649+0000 7f07708f3640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f076c141600 0x7f076c137830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:07.644 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.649+0000 7f0763fff640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f076c0b4d00 0x7f076c1410c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:07.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.649+0000 7f076bfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f076c0b1e80 0x7f076c144b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.178:35616/0 (socket says 172.21.15.178:35616) 2024-05-15T12:42:07.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.649+0000 7f076bfff640 1 -- 172.21.15.178:0/211684469 learned_addr learned my addr 172.21.15.178:0/211684469 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-05-15T12:42:07.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.649+0000 7f076bfff640 10 monclient(hunting): get_auth_request con 0x7f076c0b1e80 auth_method 0 2024-05-15T12:42:07.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.649+0000 7f076bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:07.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.649+0000 7f076bfff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:07.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.649+0000 7f076bfff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:07.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.649+0000 7f07708f3640 10 monclient(hunting): get_auth_request con 0x7f076c141600 auth_method 0 2024-05-15T12:42:07.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.649+0000 7f07708f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:07.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.649+0000 7f07708f3640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:07.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.649+0000 7f07708f3640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:07.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.650+0000 7f0763fff640 10 monclient(hunting): get_auth_request con 0x7f076c0b4d00 auth_method 0 2024-05-15T12:42:07.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.650+0000 7f0763fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:07.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.650+0000 7f0763fff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:07.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.650+0000 7f0763fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:07.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.650+0000 7f076bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:07.646 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.650+0000 7f076bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:07.646 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.650+0000 7f076bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:07.646 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.650+0000 7f07708f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:07.646 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.650+0000 7f07708f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:07.646 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.650+0000 7f07708f3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:07.646 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.650+0000 7f0763fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:07.646 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.650+0000 7f0763fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:07.646 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.650+0000 7f0763fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:07.647 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.653+0000 7f07708f3640 10 monclient(hunting): handle_auth_done global_id 4135 payload 824 2024-05-15T12:42:07.647 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.653+0000 7f07708f3640 1 -- 172.21.15.178:0/211684469 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f076c0b4d00 msgr2=0x7f076c1410c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:07.648 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.653+0000 7f07708f3640 1 --2- 172.21.15.178:0/211684469 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f076c0b4d00 0x7f076c1410c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:07.648 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.653+0000 7f07708f3640 1 -- 172.21.15.178:0/211684469 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f076c0b1e80 msgr2=0x7f076c144b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:07.648 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.653+0000 7f07708f3640 1 --2- 172.21.15.178:0/211684469 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f076c0b1e80 0x7f076c144b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:07.648 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.653+0000 7f07708f3640 10 monclient: _finish_hunting 0 2024-05-15T12:42:07.648 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.653+0000 7f07708f3640 1 monclient: found mon.noname-b 2024-05-15T12:42:07.648 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.653+0000 7f07708f3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:42:07.648 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.653+0000 7f07708f3640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2024-05-15T12:42:07.648 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.653+0000 7f07708f3640 1 -- 172.21.15.178:0/211684469 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f076c1408e0 con 0x7f076c141600 2024-05-15T12:42:07.648 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.653+0000 7f076bfff640 1 --2- 172.21.15.178:0/211684469 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f076c0b1e80 0x7f076c144b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-15T12:42:07.648 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.655+0000 7f07708f3640 1 --2- 172.21.15.178:0/211684469 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f076c141600 0x7f076c137830 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0764033000 tx=0x7f0764004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:42:07.648 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.655+0000 7f07697fa640 10 client.?.objecter ms_handle_connect 0x7f076c141600 2024-05-15T12:42:07.648 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.655+0000 7f07697fa640 10 client.?.objecter resend_mon_ops 2024-05-15T12:42:07.648 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.657+0000 7f07697fa640 1 -- 172.21.15.178:0/211684469 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0764042030 con 0x7f076c141600 2024-05-15T12:42:07.648 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.657+0000 7f07697fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:07.649 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.657+0000 7f07697fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:42:07.649 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.657+0000 7f07697fa640 10 monclient: dump: 2024-05-15T12:42:07.649 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-05-15T12:42:07.649 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:07.649 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:07.649 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:07.649 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:07.649 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2024-05-15T12:42:07.649 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:07.649 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:07.649 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:07.649 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-05-15T12:42:07.649 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.657+0000 7f07697fa640 10 monclient: _finish_auth 0 2024-05-15T12:42:07.649 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.657+0000 7f07697fa640 10 monclient: _check_auth_tickets 2024-05-15T12:42:07.649 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.657+0000 7f07697fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:42:07.650 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.657+0000 7f0772356640 5 monclient: authenticate success, global_id 4135 2024-05-15T12:42:07.650 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.657+0000 7f0772356640 10 monclient: _renew_subs 2024-05-15T12:42:07.650 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.657+0000 7f0772356640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:42:07.650 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.657+0000 7f0772356640 1 -- 172.21.15.178:0/211684469 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f076c143f50 con 0x7f076c141600 2024-05-15T12:42:07.650 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.657+0000 7f0772356640 10 client.4135.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:42:07.650 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.657+0000 7f0772356640 10 monclient: _renew_subs 2024-05-15T12:42:07.650 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.657+0000 7f0772356640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:42:07.650 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.657+0000 7f0772356640 1 -- 172.21.15.178:0/211684469 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f076c140500 con 0x7f076c141600 2024-05-15T12:42:07.650 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.657+0000 7f0772356640 1 librados: init done 2024-05-15T12:42:07.651 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.658+0000 7f0772356640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:42:07.651 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.658+0000 7f0772356640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:42:07.651 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.658+0000 7f0772356640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:42:07.651 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.658+0000 7f0772356640 1 -- 172.21.15.178:0/211684469 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f076c1452d0 con 0x7f076c141600 2024-05-15T12:42:07.651 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.659+0000 7f07697fa640 1 -- 172.21.15.178:0/211684469 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f076403a040 con 0x7f076c141600 2024-05-15T12:42:07.651 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.659+0000 7f07697fa640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:42:07.651 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.659+0000 7f07697fa640 1 -- 172.21.15.178:0/211684469 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0764039070 con 0x7f076c141600 2024-05-15T12:42:07.652 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.659+0000 7f07697fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:07.652 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.659+0000 7f07697fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:42:07.652 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.659+0000 7f07697fa640 10 monclient: dump: 2024-05-15T12:42:07.652 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-05-15T12:42:07.652 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:07.652 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:07.652 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:07.652 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:07.652 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2024-05-15T12:42:07.652 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:07.652 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:07.652 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:07.652 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-05-15T12:42:07.652 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.659+0000 7f076affd640 4 set_mon_vals no callback set 2024-05-15T12:42:07.652 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.661+0000 7f07697fa640 1 -- 172.21.15.178:0/211684469 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f0764045070 con 0x7f076c141600 2024-05-15T12:42:07.653 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-05-15T12:42:07.662+0000 7f5134c38040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-15T12:42:07.751 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.761+0000 7f07697fa640 1 -- 172.21.15.178:0/211684469 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f0764034b20 con 0x7f076c141600 2024-05-15T12:42:07.751 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.761+0000 7f07697fa640 10 client.4135.objecter ms_dispatch 0x7f076c000b90 osd_map(4..4 src has 1..4) v4 2024-05-15T12:42:07.751 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.761+0000 7f07697fa640 3 client.4135.objecter handle_osd_map got epochs [4,4] > 0 2024-05-15T12:42:07.751 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.761+0000 7f07697fa640 3 client.4135.objecter handle_osd_map decoding full epoch 4 2024-05-15T12:42:07.751 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.761+0000 7f07697fa640 20 client.4135.objecter dump_active .. 0 homeless 2024-05-15T12:42:07.751 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.761+0000 7f07697fa640 1 -- 172.21.15.178:0/211684469 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f076404c070 con 0x7f076c141600 2024-05-15T12:42:07.752 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.761+0000 7f07697fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:42:07.752 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.761+0000 7f07697fa640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:42:07.821 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.831+0000 7f0772356640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "708e4e3e-7e79-480f-9dbf-7b972c2f2811", "id": 2}] 2024-05-15T12:42:07.821 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.831+0000 7f0772356640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "708e4e3e-7e79-480f-9dbf-7b972c2f2811", "id": 2}] 2024-05-15T12:42:07.821 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.831+0000 7f0772356640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:42:07.821 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.831+0000 7f0772356640 1 -- 172.21.15.178:0/211684469 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "osd new", "uuid": "708e4e3e-7e79-480f-9dbf-7b972c2f2811", "id": 2} v 0) v1 -- 0x7f076c04f0e0 con 0x7f076c141600 2024-05-15T12:42:07.858 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-05-15T12:42:07.868+0000 7f18296bb040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-15T12:42:07.894 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.904+0000 7f07697fa640 1 -- 172.21.15.178:0/211684469 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "708e4e3e-7e79-480f-9dbf-7b972c2f2811", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f0764034e10 con 0x7f076c141600 2024-05-15T12:42:07.894 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.904+0000 7f07697fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "708e4e3e-7e79-480f-9dbf-7b972c2f2811", "id": 2}] 2024-05-15T12:42:07.894 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.904+0000 7f07697fa640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:42:07.894 INFO:teuthology.orchestra.run.smithi178.stdout:2 2024-05-15T12:42:07.895 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.904+0000 7f0772356640 10 librados: watch_flush enter 2024-05-15T12:42:07.895 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.905+0000 7f0772356640 10 librados: watch_flush exit 2024-05-15T12:42:07.896 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.905+0000 7f0772356640 20 client.4135.objecter shutdown clearing up homeless session... 2024-05-15T12:42:07.896 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.905+0000 7f0772356640 10 client.4135.objecter successfully canceled tick 2024-05-15T12:42:07.896 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.905+0000 7f0772356640 10 monclient: shutdown 2024-05-15T12:42:07.896 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.905+0000 7f0772356640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:42:07.896 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.905+0000 7f0772356640 1 -- 172.21.15.178:0/211684469 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f076c141600 msgr2=0x7f076c137830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:07.896 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.905+0000 7f0772356640 1 --2- 172.21.15.178:0/211684469 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f076c141600 0x7f076c137830 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0764033000 tx=0x7f0764004a40 comp rx=0 tx=0).stop 2024-05-15T12:42:07.896 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.905+0000 7f0772356640 1 -- 172.21.15.178:0/211684469 shutdown_connections 2024-05-15T12:42:07.896 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.905+0000 7f0772356640 1 --2- 172.21.15.178:0/211684469 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f076c141600 0x7f076c137830 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:07.896 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.905+0000 7f0772356640 1 --2- 172.21.15.178:0/211684469 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f076c0b4d00 0x7f076c1410c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:07.896 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.905+0000 7f0772356640 1 --2- 172.21.15.178:0/211684469 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f076c0b1e80 0x7f076c144b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:07.896 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.905+0000 7f0772356640 1 -- 172.21.15.178:0/211684469 >> 172.21.15.178:0/211684469 conn(0x7f076c066cc0 msgr2=0x7f076c0b76e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:42:07.896 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.905+0000 7f0772356640 1 -- 172.21.15.178:0/211684469 shutdown_connections 2024-05-15T12:42:07.896 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.905+0000 7f0772356640 1 -- 172.21.15.178:0/211684469 wait complete. 2024-05-15T12:42:07.896 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:07.905+0000 7f0772356640 1 librados: shutdown 2024-05-15T12:42:07.906 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph --cluster ceph osd new e7b5d7d7-515c-42d9-9c41-b1ca3977865e 3 2024-05-15T12:42:07.910 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-05-15T12:42:07.920+0000 7f5134c38040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-15T12:42:07.963 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-05-15T12:42:07.973+0000 7f5134c38040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-15T12:42:08.005 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.015+0000 7f53408b7640 10 monclient: get_monmap_and_config 2024-05-15T12:42:08.005 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.015+0000 7f53408b7640 10 monclient: build_initial_monmap 2024-05-15T12:42:08.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.015+0000 7f53408b7640 10 monclient: monmap: 2024-05-15T12:42:08.006 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 0 2024-05-15T12:42:08.006 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:08.006 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-05-15T12:42:08.016353+0000 2024-05-15T12:42:08.006 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-05-15T12:42:08.016353+0000 2024-05-15T12:42:08.006 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 0 (unknown) 2024-05-15T12:42:08.006 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2024-05-15T12:42:08.006 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T12:42:08.006 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T12:42:08.006 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T12:42:08.006 INFO:teuthology.orchestra.run.smithi178.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T12:42:08.006 INFO:teuthology.orchestra.run.smithi178.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T12:42:08.006 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-05-15T12:42:08.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.015+0000 7f53408b7640 1 Processor -- start 2024-05-15T12:42:08.007 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.016+0000 7f53408b7640 1 -- start start 2024-05-15T12:42:08.007 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.016+0000 7f53408b7640 10 monclient: init 2024-05-15T12:42:08.008 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.016+0000 7f53408b7640 10 monclient: _reopen_session rank -1 2024-05-15T12:42:08.008 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.016+0000 7f53408b7640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-05-15T12:42:08.008 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.016+0000 7f53408b7640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f533c0b4d00 0x7f533c0b1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:08.008 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.016+0000 7f53408b7640 10 monclient(hunting): picked mon.noname-b con 0x7f533c0b4d00 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:42:08.008 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.016+0000 7f53408b7640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f533c0b50e0 0x7f533c145c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:08.008 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.016+0000 7f53408b7640 10 monclient(hunting): picked mon.noname-a con 0x7f533c0b50e0 addr v2:172.21.15.117:3300/0 2024-05-15T12:42:08.008 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.016+0000 7f53408b7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f533c0a94b0 addr v1:172.21.15.117:6789/0 2024-05-15T12:42:08.008 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.016+0000 7f53408b7640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f533c0ab1b0 con 0x7f533c0a94b0 2024-05-15T12:42:08.008 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.016+0000 7f53408b7640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:08.008 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.016+0000 7f53408b7640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f533c0b23c0 con 0x7f533c0b50e0 2024-05-15T12:42:08.008 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.016+0000 7f53408b7640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:08.009 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.016+0000 7f53408b7640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f533c0b16f0 con 0x7f533c0b4d00 2024-05-15T12:42:08.009 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.016+0000 7f53408b7640 10 monclient(hunting): _renew_subs 2024-05-15T12:42:08.009 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.016+0000 7f53408b7640 10 monclient(hunting): authenticate will time out at 1284.354219s 2024-05-15T12:42:08.009 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.018+0000 7f533a59c640 1 -- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f533c0b4d00 msgr2=0x7f533c0b1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-15T12:42:08.009 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.018+0000 7f533a59c640 1 -- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f533c0b4d00 msgr2=0x7f533c0b1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-15T12:42:08.009 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.018+0000 7f533a59c640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f533c0b4d00 0x7f533c0b1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted) 2024-05-15T12:42:08.009 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.018+0000 7f533a59c640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f533c0b4d00 0x7f533c0b1e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-15T12:42:08.010 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.020+0000 7f5339d9b640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f533c0b50e0 0x7f533c145c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:08.011 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.021+0000 7f5339d9b640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f533c0b50e0 0x7f533c145c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.178:35632/0 (socket says 172.21.15.178:35632) 2024-05-15T12:42:08.011 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.021+0000 7f5339d9b640 1 -- 172.21.15.178:0/3251920083 learned_addr learned my addr 172.21.15.178:0/3251920083 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-05-15T12:42:08.011 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.021+0000 7f5339d9b640 10 monclient(hunting): get_auth_request con 0x7f533c0b50e0 auth_method 0 2024-05-15T12:42:08.011 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.021+0000 7f5339d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:08.011 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.021+0000 7f5339d9b640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:08.011 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.021+0000 7f5339d9b640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:08.025 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.036+0000 7f5339d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:08.026 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.036+0000 7f5339d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:08.026 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.036+0000 7f5339d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:08.034 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.044+0000 7f5339d9b640 10 monclient(hunting): handle_auth_done global_id 4131 payload 274 2024-05-15T12:42:08.034 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.044+0000 7f5339d9b640 1 -- 172.21.15.178:0/3251920083 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f533c0b4d00 msgr2=0x7f533c0b1e80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-15T12:42:08.034 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.044+0000 7f5339d9b640 1 --2- 172.21.15.178:0/3251920083 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f533c0b4d00 0x7f533c0b1e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:08.035 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.044+0000 7f5339d9b640 1 -- 172.21.15.178:0/3251920083 >> v1:172.21.15.117:6789/0 conn(0x7f533c0a94b0 legacy=0x7f533c0b0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:08.035 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.044+0000 7f5339d9b640 10 monclient: _finish_hunting 0 2024-05-15T12:42:08.035 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.044+0000 7f5339d9b640 1 monclient: found mon.noname-a 2024-05-15T12:42:08.035 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.044+0000 7f5339d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:42:08.035 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.044+0000 7f5339d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.117:3300/0 2024-05-15T12:42:08.035 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.044+0000 7f5339d9b640 1 -- 172.21.15.178:0/3251920083 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f533c146160 con 0x7f533c0b50e0 2024-05-15T12:42:08.039 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.049+0000 7f5339d9b640 1 --2- 172.21.15.178:0/3251920083 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f533c0b50e0 0x7f533c145c20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f53300098a0 tx=0x7f533002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=458967bb69cdfafa server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:42:08.044 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.054+0000 7f533959a640 1 -- 172.21.15.178:0/3251920083 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5330035020 con 0x7f533c0b50e0 2024-05-15T12:42:08.044 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.054+0000 7f533959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:08.044 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.054+0000 7f533959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T12:42:08.044 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.054+0000 7f533959a640 10 monclient: dump: 2024-05-15T12:42:08.044 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-05-15T12:42:08.045 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:08.045 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:08.045 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:08.045 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:08.045 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2024-05-15T12:42:08.045 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:08.045 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:08.045 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:08.045 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-05-15T12:42:08.045 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.054+0000 7f533959a640 10 monclient: _finish_auth 0 2024-05-15T12:42:08.045 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.054+0000 7f533959a640 10 monclient: _check_auth_tickets 2024-05-15T12:42:08.045 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.054+0000 7f533959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:42:08.045 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.054+0000 7f533959a640 1 -- 172.21.15.178:0/3251920083 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5330031640 con 0x7f533c0b50e0 2024-05-15T12:42:08.045 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.054+0000 7f533959a640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:42:08.046 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.054+0000 7f533959a640 1 -- 172.21.15.178:0/3251920083 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f533002d040 con 0x7f533c0b50e0 2024-05-15T12:42:08.046 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.054+0000 7f53408b7640 5 monclient: authenticate success, global_id 4131 2024-05-15T12:42:08.046 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.054+0000 7f53408b7640 10 monclient: get_monmap_and_config success 2024-05-15T12:42:08.046 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.054+0000 7f53408b7640 4 set_mon_vals no callback set 2024-05-15T12:42:08.046 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.054+0000 7f533959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:08.046 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.054+0000 7f533959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T12:42:08.046 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.054+0000 7f533959a640 10 monclient: dump: 2024-05-15T12:42:08.046 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-05-15T12:42:08.046 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:08.046 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:08.046 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:08.046 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:08.046 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2024-05-15T12:42:08.046 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:08.047 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:08.047 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:08.047 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-05-15T12:42:08.050 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.055+0000 7f53408b7640 10 monclient: shutdown 2024-05-15T12:42:08.050 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.055+0000 7f53408b7640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:42:08.050 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.055+0000 7f53408b7640 1 -- 172.21.15.178:0/3251920083 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f533c0b50e0 msgr2=0x7f533c145c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:08.050 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.055+0000 7f53408b7640 1 --2- 172.21.15.178:0/3251920083 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f533c0b50e0 0x7f533c145c20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f53300098a0 tx=0x7f533002b8f0 comp rx=0 tx=0).stop 2024-05-15T12:42:08.050 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.055+0000 7f53408b7640 1 -- 172.21.15.178:0/3251920083 shutdown_connections 2024-05-15T12:42:08.050 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.055+0000 7f53408b7640 1 --2- 172.21.15.178:0/3251920083 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f533c0b50e0 0x7f533c145c20 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:08.050 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.055+0000 7f53408b7640 1 --2- 172.21.15.178:0/3251920083 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f533c0b4d00 0x7f533c0b1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:08.050 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.055+0000 7f53408b7640 1 -- 172.21.15.178:0/3251920083 >> 172.21.15.178:0/3251920083 conn(0x7f533c066cc0 msgr2=0x7f533c0670d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:42:08.050 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.055+0000 7f53408b7640 1 -- 172.21.15.178:0/3251920083 shutdown_connections 2024-05-15T12:42:08.050 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.055+0000 7f53408b7640 1 -- 172.21.15.178:0/3251920083 wait complete. 2024-05-15T12:42:08.050 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.056+0000 7f53408b7640 10 monclient: build_initial_monmap 2024-05-15T12:42:08.050 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.056+0000 7f53408b7640 10 monclient: monmap: 2024-05-15T12:42:08.051 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 0 2024-05-15T12:42:08.051 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:42:08.051 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 0.000000 2024-05-15T12:42:08.051 INFO:teuthology.orchestra.run.smithi178.stderr:created 0.000000 2024-05-15T12:42:08.051 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 0 (unknown) 2024-05-15T12:42:08.051 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2024-05-15T12:42:08.051 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T12:42:08.051 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T12:42:08.051 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T12:42:08.051 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-05-15T12:42:08.051 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.056+0000 7f53408b7640 1 librados: starting msgr at 2024-05-15T12:42:08.052 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.056+0000 7f53408b7640 1 librados: starting objecter 2024-05-15T12:42:08.052 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.056+0000 7f53408b7640 1 Processor -- start 2024-05-15T12:42:08.052 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.056+0000 7f53408b7640 1 -- start start 2024-05-15T12:42:08.052 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.056+0000 7f53408b7640 1 librados: setting wanted keys 2024-05-15T12:42:08.052 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.056+0000 7f53408b7640 1 librados: calling monclient init 2024-05-15T12:42:08.052 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.056+0000 7f53408b7640 10 monclient: init 2024-05-15T12:42:08.052 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.057+0000 7f53408b7640 10 monclient: _reopen_session rank -1 2024-05-15T12:42:08.052 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.057+0000 7f53408b7640 10 monclient: _add_conns ranks=[2,0,1] 2024-05-15T12:42:08.052 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.057+0000 7f53408b7640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f533c0a94b0 0x7f533c1439b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:08.052 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.057+0000 7f53408b7640 10 monclient(hunting): picked mon.noname-c con 0x7f533c0a94b0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:42:08.052 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.057+0000 7f53408b7640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f533c0b4d00 0x7f533c1380f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:08.052 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.057+0000 7f53408b7640 10 monclient(hunting): picked mon.noname-a con 0x7f533c0b4d00 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T12:42:08.052 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.057+0000 7f53408b7640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f533c1442e0 0x7f533c138630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:08.052 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.057+0000 7f53408b7640 10 monclient(hunting): picked mon.noname-b con 0x7f533c1442e0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:42:08.052 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.057+0000 7f53408b7640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:08.053 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.057+0000 7f53408b7640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f533c14cfb0 con 0x7f533c0b4d00 2024-05-15T12:42:08.053 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.057+0000 7f53408b7640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:08.053 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.057+0000 7f53408b7640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f533c14d130 con 0x7f533c1442e0 2024-05-15T12:42:08.053 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.057+0000 7f53408b7640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:08.053 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.057+0000 7f53408b7640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f533c14d2b0 con 0x7f533c0a94b0 2024-05-15T12:42:08.053 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.057+0000 7f53408b7640 10 monclient(hunting): _renew_subs 2024-05-15T12:42:08.053 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.057+0000 7f53408b7640 10 monclient(hunting): authenticate will time out at 1284.394820s 2024-05-15T12:42:08.053 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.057+0000 7f533ad9d640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f533c1442e0 0x7f533c138630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:08.053 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.058+0000 7f533a59c640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f533c0a94b0 0x7f533c1439b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:08.053 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.058+0000 7f533ad9d640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f533c1442e0 0x7f533c138630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.178:44956/0 (socket says 172.21.15.178:44956) 2024-05-15T12:42:08.053 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.058+0000 7f533ad9d640 1 -- 172.21.15.178:0/1146932932 learned_addr learned my addr 172.21.15.178:0/1146932932 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-05-15T12:42:08.053 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.058+0000 7f533ad9d640 10 monclient(hunting): get_auth_request con 0x7f533c1442e0 auth_method 0 2024-05-15T12:42:08.053 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.058+0000 7f533ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:08.053 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.058+0000 7f533ad9d640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:08.053 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.058+0000 7f533ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:08.054 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.058+0000 7f533ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:08.054 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.058+0000 7f533ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:08.054 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.058+0000 7f533ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:08.054 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.058+0000 7f533a59c640 10 monclient(hunting): get_auth_request con 0x7f533c0a94b0 auth_method 0 2024-05-15T12:42:08.054 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.058+0000 7f533a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:08.054 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.058+0000 7f533a59c640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:08.054 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.058+0000 7f533a59c640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:08.054 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.059+0000 7f533a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:08.054 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.059+0000 7f533a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:08.054 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.059+0000 7f533a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:08.054 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.059+0000 7f5339d9b640 1 --2- 172.21.15.178:0/1146932932 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f533c0b4d00 0x7f533c1380f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:08.054 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.059+0000 7f5339d9b640 10 monclient(hunting): get_auth_request con 0x7f533c0b4d00 auth_method 0 2024-05-15T12:42:08.054 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.059+0000 7f5339d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:08.054 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.059+0000 7f5339d9b640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:08.055 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.059+0000 7f5339d9b640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:08.055 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.060+0000 7f5339d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:08.055 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.060+0000 7f5339d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:08.055 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.060+0000 7f5339d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:08.056 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.061+0000 7f533ad9d640 10 monclient(hunting): handle_auth_done global_id 4138 payload 824 2024-05-15T12:42:08.056 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.061+0000 7f533ad9d640 1 -- 172.21.15.178:0/1146932932 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f533c0a94b0 msgr2=0x7f533c1439b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:08.056 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.061+0000 7f533ad9d640 1 --2- 172.21.15.178:0/1146932932 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f533c0a94b0 0x7f533c1439b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:08.056 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.061+0000 7f533ad9d640 1 -- 172.21.15.178:0/1146932932 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f533c0b4d00 msgr2=0x7f533c1380f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:08.056 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.061+0000 7f533ad9d640 1 --2- 172.21.15.178:0/1146932932 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f533c0b4d00 0x7f533c1380f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:08.056 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.061+0000 7f533ad9d640 10 monclient: _finish_hunting 0 2024-05-15T12:42:08.056 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.061+0000 7f533ad9d640 1 monclient: found mon.noname-b 2024-05-15T12:42:08.056 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.061+0000 7f533ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:42:08.056 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.061+0000 7f533ad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2024-05-15T12:42:08.056 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.061+0000 7f533ad9d640 1 -- 172.21.15.178:0/1146932932 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f533c138b70 con 0x7f533c1442e0 2024-05-15T12:42:08.056 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.063+0000 7f533ad9d640 1 --2- 172.21.15.178:0/1146932932 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f533c1442e0 0x7f533c138630 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f533400c730 tx=0x7f533400cbf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:42:08.056 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.063+0000 7f532affd640 10 client.?.objecter ms_handle_connect 0x7f533c1442e0 2024-05-15T12:42:08.057 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.063+0000 7f532affd640 10 client.?.objecter resend_mon_ops 2024-05-15T12:42:08.057 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.067+0000 7f532affd640 1 -- 172.21.15.178:0/1146932932 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5334003b70 con 0x7f533c1442e0 2024-05-15T12:42:08.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.067+0000 7f532affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:08.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.067+0000 7f532affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:42:08.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.067+0000 7f532affd640 10 monclient: dump: 2024-05-15T12:42:08.058 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-05-15T12:42:08.058 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:08.058 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:08.058 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:08.058 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:08.058 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2024-05-15T12:42:08.058 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:08.058 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:08.058 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:08.058 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-05-15T12:42:08.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.067+0000 7f532affd640 10 monclient: _finish_auth 0 2024-05-15T12:42:08.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.067+0000 7f532affd640 10 monclient: _check_auth_tickets 2024-05-15T12:42:08.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.067+0000 7f532affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:42:08.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.067+0000 7f532affd640 1 -- 172.21.15.178:0/1146932932 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5334003d30 con 0x7f533c1442e0 2024-05-15T12:42:08.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.067+0000 7f532affd640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:42:08.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.067+0000 7f532affd640 1 -- 172.21.15.178:0/1146932932 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5334005310 con 0x7f533c1442e0 2024-05-15T12:42:08.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.067+0000 7f53408b7640 5 monclient: authenticate success, global_id 4138 2024-05-15T12:42:08.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.067+0000 7f53408b7640 10 monclient: _renew_subs 2024-05-15T12:42:08.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.067+0000 7f53408b7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:42:08.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.067+0000 7f53408b7640 1 -- 172.21.15.178:0/1146932932 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f533c144db0 con 0x7f533c1442e0 2024-05-15T12:42:08.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.067+0000 7f532affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:08.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.067+0000 7f532affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:42:08.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.067+0000 7f532affd640 10 monclient: dump: 2024-05-15T12:42:08.059 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-05-15T12:42:08.059 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:08.059 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:08.060 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:08.060 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:08.060 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2024-05-15T12:42:08.060 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:08.060 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:08.060 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:08.060 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-05-15T12:42:08.060 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.067+0000 7f5338d99640 4 set_mon_vals no callback set 2024-05-15T12:42:08.060 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.067+0000 7f53408b7640 10 client.4138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:42:08.060 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.067+0000 7f53408b7640 10 monclient: _renew_subs 2024-05-15T12:42:08.060 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.067+0000 7f53408b7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:42:08.060 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.067+0000 7f53408b7640 1 -- 172.21.15.178:0/1146932932 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f533c143ef0 con 0x7f533c1442e0 2024-05-15T12:42:08.060 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.067+0000 7f53408b7640 1 librados: init done 2024-05-15T12:42:08.061 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.068+0000 7f53408b7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:42:08.061 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.068+0000 7f53408b7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:42:08.061 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.068+0000 7f53408b7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:42:08.061 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.068+0000 7f53408b7640 1 -- 172.21.15.178:0/1146932932 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f533c1440c0 con 0x7f533c1442e0 2024-05-15T12:42:08.099 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-05-15T12:42:08.110+0000 7f5134c38040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-15T12:42:08.129 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-05-15T12:42:08.139+0000 7f18296bb040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-15T12:42:08.160 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.169+0000 7f532affd640 1 -- 172.21.15.178:0/1146932932 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f53340056c0 con 0x7f533c1442e0 2024-05-15T12:42:08.160 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.169+0000 7f532affd640 1 -- 172.21.15.178:0/1146932932 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f5334025b30 con 0x7f533c1442e0 2024-05-15T12:42:08.160 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.169+0000 7f532affd640 10 client.4138.objecter ms_dispatch 0x7f533c000b90 osd_map(5..5 src has 1..5) v4 2024-05-15T12:42:08.160 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.169+0000 7f532affd640 3 client.4138.objecter handle_osd_map got epochs [5,5] > 0 2024-05-15T12:42:08.160 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.169+0000 7f532affd640 3 client.4138.objecter handle_osd_map decoding full epoch 5 2024-05-15T12:42:08.160 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.169+0000 7f532affd640 20 client.4138.objecter dump_active .. 0 homeless 2024-05-15T12:42:08.160 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.169+0000 7f532affd640 1 -- 172.21.15.178:0/1146932932 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f5334025da0 con 0x7f533c1442e0 2024-05-15T12:42:08.160 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.169+0000 7f532affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:42:08.160 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.169+0000 7f532affd640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:42:08.184 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-05-15T12:42:08.194+0000 7f18296bb040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-15T12:42:08.234 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.244+0000 7f53408b7640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "e7b5d7d7-515c-42d9-9c41-b1ca3977865e", "id": 3}] 2024-05-15T12:42:08.235 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.244+0000 7f53408b7640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "e7b5d7d7-515c-42d9-9c41-b1ca3977865e", "id": 3}] 2024-05-15T12:42:08.235 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.244+0000 7f53408b7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:42:08.235 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.244+0000 7f53408b7640 1 -- 172.21.15.178:0/1146932932 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e7b5d7d7-515c-42d9-9c41-b1ca3977865e", "id": 3} v 0) v1 -- 0x7f533c04f0e0 con 0x7f533c1442e0 2024-05-15T12:42:08.304 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-05-15T12:42:08.314+0000 7f18296bb040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-15T12:42:08.312 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.322+0000 7f532affd640 1 -- 172.21.15.178:0/1146932932 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e7b5d7d7-515c-42d9-9c41-b1ca3977865e", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f5334025830 con 0x7f533c1442e0 2024-05-15T12:42:08.312 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.322+0000 7f532affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "e7b5d7d7-515c-42d9-9c41-b1ca3977865e", "id": 3}] 2024-05-15T12:42:08.312 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.322+0000 7f532affd640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:42:08.313 INFO:teuthology.orchestra.run.smithi178.stdout:3 2024-05-15T12:42:08.314 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.323+0000 7f53408b7640 10 librados: watch_flush enter 2024-05-15T12:42:08.314 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.323+0000 7f53408b7640 10 librados: watch_flush exit 2024-05-15T12:42:08.314 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.323+0000 7f53408b7640 20 client.4138.objecter shutdown clearing up homeless session... 2024-05-15T12:42:08.314 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.323+0000 7f53408b7640 10 client.4138.objecter successfully canceled tick 2024-05-15T12:42:08.314 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.323+0000 7f53408b7640 10 monclient: shutdown 2024-05-15T12:42:08.314 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.323+0000 7f53408b7640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:42:08.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.323+0000 7f53408b7640 1 -- 172.21.15.178:0/1146932932 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f533c1442e0 msgr2=0x7f533c138630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:08.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.323+0000 7f53408b7640 1 --2- 172.21.15.178:0/1146932932 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f533c1442e0 0x7f533c138630 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f533400c730 tx=0x7f533400cbf0 comp rx=0 tx=0).stop 2024-05-15T12:42:08.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.323+0000 7f53408b7640 1 -- 172.21.15.178:0/1146932932 shutdown_connections 2024-05-15T12:42:08.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.323+0000 7f53408b7640 1 --2- 172.21.15.178:0/1146932932 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f533c1442e0 0x7f533c138630 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:08.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.323+0000 7f53408b7640 1 --2- 172.21.15.178:0/1146932932 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f533c0b4d00 0x7f533c1380f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:08.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.323+0000 7f53408b7640 1 --2- 172.21.15.178:0/1146932932 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f533c0a94b0 0x7f533c1439b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:08.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.323+0000 7f53408b7640 1 -- 172.21.15.178:0/1146932932 >> 172.21.15.178:0/1146932932 conn(0x7f533c066cc0 msgr2=0x7f533c13e350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:42:08.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.324+0000 7f53408b7640 1 -- 172.21.15.178:0/1146932932 shutdown_connections 2024-05-15T12:42:08.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.324+0000 7f53408b7640 1 -- 172.21.15.178:0/1146932932 wait complete. 2024-05-15T12:42:08.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.324+0000 7f53408b7640 1 librados: shutdown 2024-05-15T12:42:08.324 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph --cluster ceph osd new fa647c2c-942b-4be3-afb9-6dba6b0a8cf3 4 2024-05-15T12:42:08.395 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-05-15T12:42:08.405+0000 7f5134c38040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-15T12:42:08.422 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.431+0000 7f26a7ff7640 10 monclient: get_monmap_and_config 2024-05-15T12:42:08.422 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.431+0000 7f26a7ff7640 10 monclient: build_initial_monmap 2024-05-15T12:42:08.422 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.431+0000 7f26a7ff7640 10 monclient: monmap: 2024-05-15T12:42:08.422 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 0 2024-05-15T12:42:08.422 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:08.422 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-05-15T12:42:08.432903+0000 2024-05-15T12:42:08.423 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-05-15T12:42:08.432903+0000 2024-05-15T12:42:08.423 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 0 (unknown) 2024-05-15T12:42:08.423 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2024-05-15T12:42:08.423 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T12:42:08.423 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T12:42:08.423 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T12:42:08.423 INFO:teuthology.orchestra.run.smithi178.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T12:42:08.423 INFO:teuthology.orchestra.run.smithi178.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T12:42:08.423 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-05-15T12:42:08.423 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.432+0000 7f26a7ff7640 1 Processor -- start 2024-05-15T12:42:08.423 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.432+0000 7f26a7ff7640 1 -- start start 2024-05-15T12:42:08.423 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.432+0000 7f26a7ff7640 10 monclient: init 2024-05-15T12:42:08.426 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.433+0000 7f26a7ff7640 10 monclient: _reopen_session rank -1 2024-05-15T12:42:08.426 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.433+0000 7f26a7ff7640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-05-15T12:42:08.426 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.433+0000 7f26a7ff7640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f26a00b4d00 0x7f26a00b1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:08.426 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.433+0000 7f26a7ff7640 10 monclient(hunting): picked mon.noname-a con 0x7f26a00b4d00 addr v2:172.21.15.117:3300/0 2024-05-15T12:42:08.426 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.433+0000 7f26a7ff7640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f26a00b50e0 0x7f26a0145c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:08.426 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.433+0000 7f26a7ff7640 10 monclient(hunting): picked mon.noname-b con 0x7f26a00b50e0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:42:08.426 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.433+0000 7f26a7ff7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f26a00a94b0 addr v1:172.21.15.178:6789/0 2024-05-15T12:42:08.426 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.433+0000 7f26a7ff7640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26a00ab1b0 con 0x7f26a00a94b0 2024-05-15T12:42:08.426 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.433+0000 7f26a7ff7640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:08.426 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.433+0000 7f26a7ff7640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26a00b23c0 con 0x7f26a00b4d00 2024-05-15T12:42:08.426 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.433+0000 7f26a7ff7640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:08.426 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.433+0000 7f26a7ff7640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26a00b16f0 con 0x7f26a00b50e0 2024-05-15T12:42:08.426 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.433+0000 7f26a7ff7640 10 monclient(hunting): _renew_subs 2024-05-15T12:42:08.427 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.433+0000 7f26a7ff7640 10 monclient(hunting): authenticate will time out at 1284.770741s 2024-05-15T12:42:08.427 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.433+0000 7f26a6594640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f26a00a94b0 0x7f26a00b0fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.178:60872/0 (socket says 172.21.15.178:60872) 2024-05-15T12:42:08.427 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.433+0000 7f26a6594640 1 -- 172.21.15.178:0/1126593506 learned_addr learned my addr 172.21.15.178:0/1126593506 (peer_addr_for_me v1:172.21.15.178:0/0) 2024-05-15T12:42:08.427 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.434+0000 7f26a5d93640 1 --2- 172.21.15.178:0/1126593506 >> v2:172.21.15.117:3300/0 conn(0x7f26a00b4d00 0x7f26a00b1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:08.427 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.434+0000 7f26a5592640 1 --2- 172.21.15.178:0/1126593506 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f26a00b50e0 0x7f26a0145c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:08.427 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.434+0000 7f26a5d93640 10 monclient(hunting): get_auth_request con 0x7f26a00b4d00 auth_method 0 2024-05-15T12:42:08.427 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.434+0000 7f26a5d93640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:08.427 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.434+0000 7f26a5d93640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:08.427 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.434+0000 7f26a5d93640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:08.427 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.434+0000 7f26a5592640 10 monclient(hunting): get_auth_request con 0x7f26a00b50e0 auth_method 0 2024-05-15T12:42:08.427 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.434+0000 7f26a5592640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:08.427 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.434+0000 7f26a5592640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:08.427 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.434+0000 7f26a5592640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:08.427 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.434+0000 7f26a5d93640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:08.428 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.434+0000 7f26a5d93640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:08.428 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.434+0000 7f26a5d93640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:08.428 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.435+0000 7f26a5592640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:08.428 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.435+0000 7f26a5592640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:08.428 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.435+0000 7f26a5592640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:08.429 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.436+0000 7f26a5d93640 10 monclient(hunting): handle_auth_done global_id 4140 payload 274 2024-05-15T12:42:08.429 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.436+0000 7f26a5d93640 1 -- 172.21.15.178:0/1126593506 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f26a00b50e0 msgr2=0x7f26a0145c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:42:08.429 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.436+0000 7f26a5d93640 1 --2- 172.21.15.178:0/1126593506 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f26a00b50e0 0x7f26a0145c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:08.429 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.436+0000 7f26a5d93640 1 -- 172.21.15.178:0/1126593506 >> v1:172.21.15.178:6789/0 conn(0x7f26a00a94b0 legacy=0x7f26a00b0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:08.429 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.436+0000 7f26a5d93640 10 monclient: _finish_hunting 0 2024-05-15T12:42:08.429 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.436+0000 7f26a5d93640 1 monclient: found mon.noname-a 2024-05-15T12:42:08.429 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.436+0000 7f26a5d93640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:42:08.429 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.436+0000 7f26a5d93640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.117:3300/0 2024-05-15T12:42:08.429 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.436+0000 7f26a5d93640 1 -- 172.21.15.178:0/1126593506 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26a0146160 con 0x7f26a00b4d00 2024-05-15T12:42:08.429 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.438+0000 7f26a5d93640 1 --2- 172.21.15.178:0/1126593506 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f26a00b4d00 0x7f26a00b1e80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f269c0097d0 tx=0x7f269c02b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c652642c7d171648 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:42:08.430 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.440+0000 7f26a4d91640 1 -- 172.21.15.178:0/1126593506 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f269c031770 con 0x7f26a00b4d00 2024-05-15T12:42:08.430 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.440+0000 7f26a4d91640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:08.431 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.440+0000 7f26a4d91640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T12:42:08.431 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.440+0000 7f26a4d91640 10 monclient: dump: 2024-05-15T12:42:08.431 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-05-15T12:42:08.431 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:08.431 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:08.431 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:08.431 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:08.431 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2024-05-15T12:42:08.431 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:08.431 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:08.431 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:08.431 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-05-15T12:42:08.431 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.440+0000 7f26a4d91640 10 monclient: _finish_auth 0 2024-05-15T12:42:08.432 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.440+0000 7f26a4d91640 10 monclient: _check_auth_tickets 2024-05-15T12:42:08.432 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.440+0000 7f26a4d91640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:42:08.432 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.440+0000 7f26a7ff7640 5 monclient: authenticate success, global_id 4140 2024-05-15T12:42:08.432 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.440+0000 7f26a7ff7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:42:08.434 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.442+0000 7f26a4d91640 1 -- 172.21.15.178:0/1126593506 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f269c037070 con 0x7f26a00b4d00 2024-05-15T12:42:08.434 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.442+0000 7f26a4d91640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:42:08.434 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.442+0000 7f26a4d91640 1 -- 172.21.15.178:0/1126593506 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f269c031b30 con 0x7f26a00b4d00 2024-05-15T12:42:08.434 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.442+0000 7f26a4d91640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:08.434 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.442+0000 7f26a4d91640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T12:42:08.434 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.442+0000 7f26a4d91640 10 monclient: dump: 2024-05-15T12:42:08.434 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-05-15T12:42:08.434 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:08.434 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:08.434 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:08.434 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:08.435 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2024-05-15T12:42:08.435 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:08.435 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:08.435 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:08.435 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-05-15T12:42:08.435 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.442+0000 7f26a7ff7640 10 monclient: get_monmap_and_config success 2024-05-15T12:42:08.435 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.442+0000 7f26a7ff7640 4 set_mon_vals no callback set 2024-05-15T12:42:08.435 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.443+0000 7f26a7ff7640 10 monclient: shutdown 2024-05-15T12:42:08.435 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.443+0000 7f26a7ff7640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:42:08.435 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.443+0000 7f26a7ff7640 1 -- 172.21.15.178:0/1126593506 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f26a00b4d00 msgr2=0x7f26a00b1e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:08.435 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.443+0000 7f26a7ff7640 1 --2- 172.21.15.178:0/1126593506 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f26a00b4d00 0x7f26a00b1e80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f269c0097d0 tx=0x7f269c02b810 comp rx=0 tx=0).stop 2024-05-15T12:42:08.435 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.443+0000 7f26a7ff7640 1 -- 172.21.15.178:0/1126593506 shutdown_connections 2024-05-15T12:42:08.435 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.443+0000 7f26a7ff7640 1 --2- 172.21.15.178:0/1126593506 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f26a00b50e0 0x7f26a0145c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:08.435 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.443+0000 7f26a7ff7640 1 --2- 172.21.15.178:0/1126593506 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f26a00b4d00 0x7f26a00b1e80 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:08.435 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.443+0000 7f26a7ff7640 1 -- 172.21.15.178:0/1126593506 >> 172.21.15.178:0/1126593506 conn(0x7f26a0066cc0 msgr2=0x7f26a00670d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:42:08.436 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.443+0000 7f26a7ff7640 1 -- 172.21.15.178:0/1126593506 shutdown_connections 2024-05-15T12:42:08.436 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.443+0000 7f26a7ff7640 1 -- 172.21.15.178:0/1126593506 wait complete. 2024-05-15T12:42:08.436 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.443+0000 7f26a7ff7640 10 monclient: build_initial_monmap 2024-05-15T12:42:08.436 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.443+0000 7f26a7ff7640 10 monclient: monmap: 2024-05-15T12:42:08.436 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 0 2024-05-15T12:42:08.436 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:42:08.436 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 0.000000 2024-05-15T12:42:08.436 INFO:teuthology.orchestra.run.smithi178.stderr:created 0.000000 2024-05-15T12:42:08.436 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 0 (unknown) 2024-05-15T12:42:08.436 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2024-05-15T12:42:08.436 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T12:42:08.436 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T12:42:08.436 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T12:42:08.436 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-05-15T12:42:08.436 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.444+0000 7f26a7ff7640 1 librados: starting msgr at 2024-05-15T12:42:08.436 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.444+0000 7f26a7ff7640 1 librados: starting objecter 2024-05-15T12:42:08.437 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.444+0000 7f26a7ff7640 1 Processor -- start 2024-05-15T12:42:08.437 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.444+0000 7f26a7ff7640 1 -- start start 2024-05-15T12:42:08.437 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.444+0000 7f26a7ff7640 1 librados: setting wanted keys 2024-05-15T12:42:08.437 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.444+0000 7f26a7ff7640 1 librados: calling monclient init 2024-05-15T12:42:08.437 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.444+0000 7f26a7ff7640 10 monclient: init 2024-05-15T12:42:08.439 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.444+0000 7f26a7ff7640 10 monclient: _reopen_session rank -1 2024-05-15T12:42:08.440 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.444+0000 7f26a7ff7640 10 monclient: _add_conns ranks=[2,0,1] 2024-05-15T12:42:08.440 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.444+0000 7f26a7ff7640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f26a00a94b0 0x7f26a0142a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:08.440 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.444+0000 7f26a7ff7640 10 monclient(hunting): picked mon.noname-c con 0x7f26a00a94b0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:42:08.440 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.444+0000 7f26a7ff7640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f26a00b4d00 0x7f26a013c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:08.440 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.444+0000 7f26a7ff7640 10 monclient(hunting): picked mon.noname-a con 0x7f26a00b4d00 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T12:42:08.440 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.444+0000 7f26a7ff7640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f26a013ea70 0x7f26a013dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:08.440 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.444+0000 7f26a7ff7640 10 monclient(hunting): picked mon.noname-b con 0x7f26a013ea70 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:42:08.440 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.444+0000 7f26a7ff7640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:08.440 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.444+0000 7f26a7ff7640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a014cfb0 con 0x7f26a00b4d00 2024-05-15T12:42:08.440 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.444+0000 7f26a7ff7640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:08.440 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.444+0000 7f26a7ff7640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a014d130 con 0x7f26a013ea70 2024-05-15T12:42:08.440 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.444+0000 7f26a7ff7640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:08.440 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.444+0000 7f26a7ff7640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26a014d2b0 con 0x7f26a00a94b0 2024-05-15T12:42:08.440 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.444+0000 7f26a7ff7640 10 monclient(hunting): _renew_subs 2024-05-15T12:42:08.440 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.444+0000 7f26a7ff7640 10 monclient(hunting): authenticate will time out at 1284.782226s 2024-05-15T12:42:08.440 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.445+0000 7f26a5592640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f26a00b4d00 0x7f26a013c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:08.441 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.445+0000 7f26a6594640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f26a013ea70 0x7f26a013dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:08.441 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.445+0000 7f26a5d93640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f26a00a94b0 0x7f26a0142a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:08.441 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.445+0000 7f26a5592640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f26a00b4d00 0x7f26a013c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.178:35662/0 (socket says 172.21.15.178:35662) 2024-05-15T12:42:08.441 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.445+0000 7f26a5592640 1 -- 172.21.15.178:0/3233009989 learned_addr learned my addr 172.21.15.178:0/3233009989 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-05-15T12:42:08.441 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.445+0000 7f26a5592640 10 monclient(hunting): get_auth_request con 0x7f26a00b4d00 auth_method 0 2024-05-15T12:42:08.441 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.445+0000 7f26a5592640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:08.441 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.445+0000 7f26a5592640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:08.441 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.445+0000 7f26a5592640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:08.441 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.445+0000 7f26a6594640 10 monclient(hunting): get_auth_request con 0x7f26a013ea70 auth_method 0 2024-05-15T12:42:08.441 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.445+0000 7f26a6594640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:08.441 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.445+0000 7f26a6594640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:08.441 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.445+0000 7f26a6594640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:08.441 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.445+0000 7f26a5d93640 10 monclient(hunting): get_auth_request con 0x7f26a00a94b0 auth_method 0 2024-05-15T12:42:08.441 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.446+0000 7f26a5d93640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:08.441 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.446+0000 7f26a5d93640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:08.441 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.446+0000 7f26a5d93640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:08.442 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.446+0000 7f26a5592640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:08.442 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.446+0000 7f26a5592640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:08.442 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.446+0000 7f26a5592640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:08.442 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.446+0000 7f26a6594640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:08.442 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.446+0000 7f26a6594640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:08.442 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.446+0000 7f26a6594640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:08.442 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.446+0000 7f26a5d93640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:08.442 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.446+0000 7f26a5d93640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:08.442 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.446+0000 7f26a5d93640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:08.442 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.449+0000 7f26a5592640 10 monclient(hunting): handle_auth_done global_id 4143 payload 824 2024-05-15T12:42:08.442 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.449+0000 7f26a5592640 1 -- 172.21.15.178:0/3233009989 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f26a00a94b0 msgr2=0x7f26a0142a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:08.442 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.449+0000 7f26a5592640 1 --2- 172.21.15.178:0/3233009989 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f26a00a94b0 0x7f26a0142a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:08.442 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.449+0000 7f26a5592640 1 -- 172.21.15.178:0/3233009989 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f26a013ea70 msgr2=0x7f26a013dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:08.442 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.449+0000 7f26a5592640 1 --2- 172.21.15.178:0/3233009989 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f26a013ea70 0x7f26a013dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:08.442 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.449+0000 7f26a5592640 10 monclient: _finish_hunting 0 2024-05-15T12:42:08.443 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.449+0000 7f26a5592640 1 monclient: found mon.noname-a 2024-05-15T12:42:08.443 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.449+0000 7f26a5592640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:42:08.443 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.449+0000 7f26a5592640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.117:3300/0 2024-05-15T12:42:08.443 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.449+0000 7f26a5592640 1 -- 172.21.15.178:0/3233009989 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26a013c880 con 0x7f26a00b4d00 2024-05-15T12:42:08.443 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.451+0000 7f26a5592640 1 --2- 172.21.15.178:0/3233009989 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f26a00b4d00 0x7f26a013c340 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f269400e870 tx=0x7f269400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:42:08.443 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.451+0000 7f26927fc640 10 client.?.objecter ms_handle_connect 0x7f26a00b4d00 2024-05-15T12:42:08.443 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.451+0000 7f26927fc640 10 client.?.objecter resend_mon_ops 2024-05-15T12:42:08.444 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-05-15T12:42:08.454+0000 7f5134c38040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-15T12:42:08.446 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.455+0000 7f26927fc640 1 -- 172.21.15.178:0/3233009989 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2694017070 con 0x7f26a00b4d00 2024-05-15T12:42:08.446 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.455+0000 7f26927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:08.446 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.455+0000 7f26927fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T12:42:08.446 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.455+0000 7f26927fc640 10 monclient: dump: 2024-05-15T12:42:08.446 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-05-15T12:42:08.446 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:08.446 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:08.446 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:08.446 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:08.446 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2024-05-15T12:42:08.446 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:08.446 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:08.446 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:08.447 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-05-15T12:42:08.447 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.455+0000 7f26927fc640 10 monclient: _finish_auth 0 2024-05-15T12:42:08.447 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.455+0000 7f26927fc640 10 monclient: _check_auth_tickets 2024-05-15T12:42:08.447 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.455+0000 7f26927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:42:08.447 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.455+0000 7f26927fc640 1 -- 172.21.15.178:0/3233009989 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f269400c940 con 0x7f26a00b4d00 2024-05-15T12:42:08.447 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.455+0000 7f26a7ff7640 5 monclient: authenticate success, global_id 4143 2024-05-15T12:42:08.447 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.455+0000 7f26a7ff7640 10 monclient: _renew_subs 2024-05-15T12:42:08.447 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.455+0000 7f26a7ff7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:42:08.447 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.455+0000 7f26a7ff7640 1 -- 172.21.15.178:0/3233009989 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26a00ac2b0 con 0x7f26a00b4d00 2024-05-15T12:42:08.447 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.455+0000 7f26927fc640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:42:08.447 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.455+0000 7f26927fc640 1 -- 172.21.15.178:0/3233009989 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f269400cd30 con 0x7f26a00b4d00 2024-05-15T12:42:08.447 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.455+0000 7f26927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:08.447 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.455+0000 7f26927fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T12:42:08.447 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.455+0000 7f26927fc640 10 monclient: dump: 2024-05-15T12:42:08.447 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-05-15T12:42:08.448 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:08.448 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:08.448 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:08.448 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:08.448 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2024-05-15T12:42:08.448 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:08.448 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:08.448 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:08.448 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-05-15T12:42:08.448 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.455+0000 7f2693fff640 4 set_mon_vals no callback set 2024-05-15T12:42:08.448 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.455+0000 7f26a7ff7640 10 client.4143.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:42:08.448 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.455+0000 7f26a7ff7640 10 monclient: _renew_subs 2024-05-15T12:42:08.448 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.455+0000 7f26a7ff7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:42:08.448 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.455+0000 7f26a7ff7640 1 -- 172.21.15.178:0/3233009989 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26a013eef0 con 0x7f26a00b4d00 2024-05-15T12:42:08.449 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.455+0000 7f26a7ff7640 1 librados: init done 2024-05-15T12:42:08.449 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.456+0000 7f26a7ff7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:42:08.449 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.456+0000 7f26a7ff7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:42:08.449 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.456+0000 7f26a7ff7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:42:08.449 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.456+0000 7f26a7ff7640 1 -- 172.21.15.178:0/3233009989 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26a004f0e0 con 0x7f26a00b4d00 2024-05-15T12:42:08.449 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.458+0000 7f26927fc640 1 -- 172.21.15.178:0/3233009989 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f269401a070 con 0x7f26a00b4d00 2024-05-15T12:42:08.450 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.458+0000 7f26927fc640 1 -- 172.21.15.178:0/3233009989 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f2694013470 con 0x7f26a00b4d00 2024-05-15T12:42:08.450 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.458+0000 7f26927fc640 10 client.4143.objecter ms_dispatch 0x7f26a0000b90 osd_map(6..6 src has 1..6) v4 2024-05-15T12:42:08.450 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.458+0000 7f26927fc640 3 client.4143.objecter handle_osd_map got epochs [6,6] > 0 2024-05-15T12:42:08.450 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.458+0000 7f26927fc640 3 client.4143.objecter handle_osd_map decoding full epoch 6 2024-05-15T12:42:08.450 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.458+0000 7f26927fc640 20 client.4143.objecter dump_active .. 0 homeless 2024-05-15T12:42:08.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.555+0000 7f26927fc640 1 -- 172.21.15.178:0/3233009989 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f2694006440 con 0x7f26a00b4d00 2024-05-15T12:42:08.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.555+0000 7f26927fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:42:08.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.555+0000 7f26927fc640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:42:08.602 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-05-15T12:42:08.612+0000 7f5134c38040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-15T12:42:08.604 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-05-15T12:42:08.614+0000 7f18296bb040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-15T12:42:08.617 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.626+0000 7f26a7ff7640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "fa647c2c-942b-4be3-afb9-6dba6b0a8cf3", "id": 4}] 2024-05-15T12:42:08.617 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.626+0000 7f26a7ff7640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "fa647c2c-942b-4be3-afb9-6dba6b0a8cf3", "id": 4}] 2024-05-15T12:42:08.617 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.626+0000 7f26a7ff7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:42:08.617 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.626+0000 7f26a7ff7640 1 -- 172.21.15.178:0/3233009989 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd new", "uuid": "fa647c2c-942b-4be3-afb9-6dba6b0a8cf3", "id": 4} v 0) v1 -- 0x7f26a00b72d0 con 0x7f26a00b4d00 2024-05-15T12:42:08.654 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-05-15T12:42:08.664+0000 7f18296bb040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-15T12:42:08.665 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.675+0000 7f26927fc640 1 -- 172.21.15.178:0/3233009989 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "fa647c2c-942b-4be3-afb9-6dba6b0a8cf3", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f2694006620 con 0x7f26a00b4d00 2024-05-15T12:42:08.665 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.675+0000 7f26927fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "fa647c2c-942b-4be3-afb9-6dba6b0a8cf3", "id": 4}] 2024-05-15T12:42:08.665 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.675+0000 7f26927fc640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:42:08.665 INFO:teuthology.orchestra.run.smithi178.stdout:4 2024-05-15T12:42:08.667 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.676+0000 7f26a7ff7640 10 librados: watch_flush enter 2024-05-15T12:42:08.667 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.676+0000 7f26a7ff7640 10 librados: watch_flush exit 2024-05-15T12:42:08.667 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.676+0000 7f26a7ff7640 20 client.4143.objecter shutdown clearing up homeless session... 2024-05-15T12:42:08.667 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.676+0000 7f26a7ff7640 10 client.4143.objecter successfully canceled tick 2024-05-15T12:42:08.668 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.676+0000 7f26a7ff7640 10 monclient: shutdown 2024-05-15T12:42:08.668 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.676+0000 7f26a7ff7640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:42:08.668 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.676+0000 7f26a7ff7640 1 -- 172.21.15.178:0/3233009989 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f26a00b4d00 msgr2=0x7f26a013c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:08.668 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.676+0000 7f26a7ff7640 1 --2- 172.21.15.178:0/3233009989 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f26a00b4d00 0x7f26a013c340 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f269400e870 tx=0x7f269400ed30 comp rx=0 tx=0).stop 2024-05-15T12:42:08.668 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.676+0000 7f26a7ff7640 1 -- 172.21.15.178:0/3233009989 shutdown_connections 2024-05-15T12:42:08.668 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.676+0000 7f26a7ff7640 1 --2- 172.21.15.178:0/3233009989 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f26a013ea70 0x7f26a013dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:08.668 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.676+0000 7f26a7ff7640 1 --2- 172.21.15.178:0/3233009989 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f26a00b4d00 0x7f26a013c340 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:08.668 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.676+0000 7f26a7ff7640 1 --2- 172.21.15.178:0/3233009989 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f26a00a94b0 0x7f26a0142a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:08.668 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.676+0000 7f26a7ff7640 1 -- 172.21.15.178:0/3233009989 >> 172.21.15.178:0/3233009989 conn(0x7f26a0066cc0 msgr2=0x7f26a0074610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:42:08.668 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.676+0000 7f26a7ff7640 1 -- 172.21.15.178:0/3233009989 shutdown_connections 2024-05-15T12:42:08.668 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.676+0000 7f26a7ff7640 1 -- 172.21.15.178:0/3233009989 wait complete. 2024-05-15T12:42:08.668 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.676+0000 7f26a7ff7640 1 librados: shutdown 2024-05-15T12:42:08.678 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph --cluster ceph osd new c5f9bc37-4bd6-4643-8f0d-e53c5f2c8782 5 2024-05-15T12:42:08.755 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-05-15T12:42:08.765+0000 7f5134c38040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-15T12:42:08.777 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.787+0000 7fa3d1209640 10 monclient: get_monmap_and_config 2024-05-15T12:42:08.777 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.787+0000 7fa3d1209640 10 monclient: build_initial_monmap 2024-05-15T12:42:08.777 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.787+0000 7fa3d1209640 10 monclient: monmap: 2024-05-15T12:42:08.777 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 0 2024-05-15T12:42:08.777 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:08.777 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-05-15T12:42:08.788065+0000 2024-05-15T12:42:08.777 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-05-15T12:42:08.788065+0000 2024-05-15T12:42:08.777 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 0 (unknown) 2024-05-15T12:42:08.777 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2024-05-15T12:42:08.777 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T12:42:08.777 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T12:42:08.778 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T12:42:08.778 INFO:teuthology.orchestra.run.smithi178.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T12:42:08.778 INFO:teuthology.orchestra.run.smithi178.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T12:42:08.778 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-05-15T12:42:08.780 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.787+0000 7fa3d1209640 1 Processor -- start 2024-05-15T12:42:08.780 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.787+0000 7fa3d1209640 1 -- start start 2024-05-15T12:42:08.780 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.787+0000 7fa3d1209640 10 monclient: init 2024-05-15T12:42:08.780 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.788+0000 7fa3d1209640 10 monclient: _reopen_session rank -1 2024-05-15T12:42:08.780 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.788+0000 7fa3d1209640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-05-15T12:42:08.780 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.788+0000 7fa3d1209640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa3cc0b4d00 addr v1:172.21.15.178:6789/0 2024-05-15T12:42:08.780 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.788+0000 7fa3d1209640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa3cc0b50e0 0x7fa3cc0b1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:08.781 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.788+0000 7fa3d1209640 10 monclient(hunting): picked mon.noname-b con 0x7fa3cc0b50e0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:42:08.781 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.788+0000 7fa3d1209640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fa3cc0a94b0 0x7fa3cc0b0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:08.781 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.788+0000 7fa3d1209640 10 monclient(hunting): picked mon.noname-c con 0x7fa3cc0a94b0 addr v2:172.21.15.178:3300/0 2024-05-15T12:42:08.781 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.788+0000 7fa3d1209640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3cc0ab1b0 con 0x7fa3cc0b4d00 2024-05-15T12:42:08.781 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.788+0000 7fa3d1209640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:08.781 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.788+0000 7fa3d1209640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3cc0b23c0 con 0x7fa3cc0a94b0 2024-05-15T12:42:08.781 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.788+0000 7fa3d1209640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:08.781 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.788+0000 7fa3d1209640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3cc0b1520 con 0x7fa3cc0b50e0 2024-05-15T12:42:08.781 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.788+0000 7fa3d1209640 10 monclient(hunting): _renew_subs 2024-05-15T12:42:08.781 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.788+0000 7fa3d1209640 10 monclient(hunting): authenticate will time out at 1285.125861s 2024-05-15T12:42:08.781 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.789+0000 7fa3cb59e640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fa3cc0a94b0 0x7fa3cc0b0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:08.781 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.789+0000 7fa3c3fff640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa3cc0b50e0 0x7fa3cc0b1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:08.781 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.789+0000 7fa3c3fff640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa3cc0b50e0 0x7fa3cc0b1e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.178:53706/0 (socket says 172.21.15.178:53706) 2024-05-15T12:42:08.781 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.789+0000 7fa3c3fff640 1 -- 172.21.15.178:0/531373358 learned_addr learned my addr 172.21.15.178:0/531373358 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-05-15T12:42:08.782 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.789+0000 7fa3c3fff640 10 monclient(hunting): get_auth_request con 0x7fa3cc0b50e0 auth_method 0 2024-05-15T12:42:08.782 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.789+0000 7fa3c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:08.782 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.789+0000 7fa3c3fff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:08.782 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.789+0000 7fa3c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:08.782 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.789+0000 7fa3cb59e640 10 monclient(hunting): get_auth_request con 0x7fa3cc0a94b0 auth_method 0 2024-05-15T12:42:08.782 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.789+0000 7fa3cb59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:08.782 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.789+0000 7fa3cb59e640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:08.782 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.789+0000 7fa3cb59e640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:08.782 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.790+0000 7fa3c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:08.782 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.790+0000 7fa3c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:08.782 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.790+0000 7fa3c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:08.782 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.790+0000 7fa3cb59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:08.782 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.790+0000 7fa3cb59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:08.782 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.790+0000 7fa3cb59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:08.783 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.791+0000 7fa3c3fff640 10 monclient(hunting): handle_auth_done global_id 4121 payload 274 2024-05-15T12:42:08.784 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.791+0000 7fa3c3fff640 1 -- 172.21.15.178:0/531373358 >> v2:172.21.15.178:3300/0 conn(0x7fa3cc0a94b0 msgr2=0x7fa3cc0b0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:42:08.784 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.791+0000 7fa3c3fff640 1 --2- 172.21.15.178:0/531373358 >> v2:172.21.15.178:3300/0 conn(0x7fa3cc0a94b0 0x7fa3cc0b0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:08.784 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.791+0000 7fa3c3fff640 1 -- 172.21.15.178:0/531373358 >> v1:172.21.15.178:6789/0 conn(0x7fa3cc0b4d00 legacy=0x7fa3cc145c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:08.784 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.791+0000 7fa3c3fff640 10 monclient: _finish_hunting 0 2024-05-15T12:42:08.784 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.791+0000 7fa3c3fff640 1 monclient: found mon.noname-b 2024-05-15T12:42:08.784 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.791+0000 7fa3c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:42:08.784 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.791+0000 7fa3c3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.117:3301/0 2024-05-15T12:42:08.784 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.791+0000 7fa3c3fff640 1 -- 172.21.15.178:0/531373358 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3cc0b16d0 con 0x7fa3cc0b50e0 2024-05-15T12:42:08.784 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.793+0000 7fa3c3fff640 1 --2- 172.21.15.178:0/531373358 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa3cc0b50e0 0x7fa3cc0b1e80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa3b4009870 tx=0x7fa3b402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=829227f488b3fba7 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:42:08.789 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.798+0000 7fa3ca59c640 1 -- 172.21.15.178:0/531373358 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3b4035020 con 0x7fa3cc0b50e0 2024-05-15T12:42:08.789 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.798+0000 7fa3ca59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:08.789 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.798+0000 7fa3ca59c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:42:08.789 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.798+0000 7fa3ca59c640 10 monclient: dump: 2024-05-15T12:42:08.790 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-05-15T12:42:08.790 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:08.790 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:08.790 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:08.790 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:08.790 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2024-05-15T12:42:08.790 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:08.790 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:08.790 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:08.790 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-05-15T12:42:08.790 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.798+0000 7fa3ca59c640 10 monclient: _finish_auth 0 2024-05-15T12:42:08.790 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.798+0000 7fa3ca59c640 10 monclient: _check_auth_tickets 2024-05-15T12:42:08.790 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.798+0000 7fa3ca59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:42:08.790 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.798+0000 7fa3ca59c640 1 -- 172.21.15.178:0/531373358 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3b4031640 con 0x7fa3cc0b50e0 2024-05-15T12:42:08.791 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.798+0000 7fa3d1209640 5 monclient: authenticate success, global_id 4121 2024-05-15T12:42:08.791 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.798+0000 7fa3d1209640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:42:08.791 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.798+0000 7fa3ca59c640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:42:08.791 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.798+0000 7fa3d1209640 10 monclient: get_monmap_and_config success 2024-05-15T12:42:08.791 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.798+0000 7fa3ca59c640 1 -- 172.21.15.178:0/531373358 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3b402d040 con 0x7fa3cc0b50e0 2024-05-15T12:42:08.791 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.798+0000 7fa3d1209640 4 set_mon_vals no callback set 2024-05-15T12:42:08.791 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.798+0000 7fa3ca59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:08.791 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.798+0000 7fa3ca59c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T12:42:08.791 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.798+0000 7fa3ca59c640 10 monclient: dump: 2024-05-15T12:42:08.791 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-05-15T12:42:08.791 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:08.791 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:08.791 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:08.791 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:08.792 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2024-05-15T12:42:08.792 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:08.792 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:08.792 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:08.792 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-05-15T12:42:08.792 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.799+0000 7fa3d1209640 10 monclient: shutdown 2024-05-15T12:42:08.792 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.799+0000 7fa3d1209640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:42:08.792 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.799+0000 7fa3d1209640 1 -- 172.21.15.178:0/531373358 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa3cc0b50e0 msgr2=0x7fa3cc0b1e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:08.792 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.799+0000 7fa3d1209640 1 --2- 172.21.15.178:0/531373358 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa3cc0b50e0 0x7fa3cc0b1e80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa3b4009870 tx=0x7fa3b402b8f0 comp rx=0 tx=0).stop 2024-05-15T12:42:08.792 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.799+0000 7fa3d1209640 1 -- 172.21.15.178:0/531373358 shutdown_connections 2024-05-15T12:42:08.792 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.799+0000 7fa3d1209640 1 --2- 172.21.15.178:0/531373358 >> v2:172.21.15.178:3300/0 conn(0x7fa3cc0a94b0 0x7fa3cc0b0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:08.792 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.799+0000 7fa3d1209640 1 --2- 172.21.15.178:0/531373358 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa3cc0b50e0 0x7fa3cc0b1e80 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:08.792 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.799+0000 7fa3d1209640 1 -- 172.21.15.178:0/531373358 >> 172.21.15.178:0/531373358 conn(0x7fa3cc066cc0 msgr2=0x7fa3cc0670d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:42:08.792 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.799+0000 7fa3d1209640 1 -- 172.21.15.178:0/531373358 shutdown_connections 2024-05-15T12:42:08.792 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.799+0000 7fa3d1209640 1 -- 172.21.15.178:0/531373358 wait complete. 2024-05-15T12:42:08.796 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.800+0000 7fa3d1209640 10 monclient: build_initial_monmap 2024-05-15T12:42:08.796 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.800+0000 7fa3d1209640 10 monclient: monmap: 2024-05-15T12:42:08.796 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 0 2024-05-15T12:42:08.796 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:42:08.796 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 0.000000 2024-05-15T12:42:08.796 INFO:teuthology.orchestra.run.smithi178.stderr:created 0.000000 2024-05-15T12:42:08.796 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 0 (unknown) 2024-05-15T12:42:08.796 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2024-05-15T12:42:08.796 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T12:42:08.797 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T12:42:08.797 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T12:42:08.797 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-05-15T12:42:08.797 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.800+0000 7fa3d1209640 1 librados: starting msgr at 2024-05-15T12:42:08.797 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.800+0000 7fa3d1209640 1 librados: starting objecter 2024-05-15T12:42:08.797 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.800+0000 7fa3d1209640 1 Processor -- start 2024-05-15T12:42:08.797 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.800+0000 7fa3d1209640 1 -- start start 2024-05-15T12:42:08.797 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.800+0000 7fa3d1209640 1 librados: setting wanted keys 2024-05-15T12:42:08.797 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.800+0000 7fa3d1209640 1 librados: calling monclient init 2024-05-15T12:42:08.797 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.800+0000 7fa3d1209640 10 monclient: init 2024-05-15T12:42:08.797 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.801+0000 7fa3d1209640 10 monclient: _reopen_session rank -1 2024-05-15T12:42:08.797 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.801+0000 7fa3d1209640 10 monclient: _add_conns ranks=[0,1,2] 2024-05-15T12:42:08.797 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.801+0000 7fa3d1209640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa3cc0a94b0 0x7fa3cc143560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:08.797 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.801+0000 7fa3d1209640 10 monclient(hunting): picked mon.noname-a con 0x7fa3cc0a94b0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T12:42:08.797 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.801+0000 7fa3d1209640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa3cc0b4d00 0x7fa3cc1d8180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:08.797 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.801+0000 7fa3d1209640 10 monclient(hunting): picked mon.noname-b con 0x7fa3cc0b4d00 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:42:08.798 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.801+0000 7fa3d1209640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa3cc143aa0 0x7fa3cc1dafe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:08.798 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.801+0000 7fa3d1209640 10 monclient(hunting): picked mon.noname-c con 0x7fa3cc143aa0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:42:08.798 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.801+0000 7fa3d1209640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:08.798 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.801+0000 7fa3d1209640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3cc14cfb0 con 0x7fa3cc0a94b0 2024-05-15T12:42:08.798 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.801+0000 7fa3d1209640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:08.798 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.801+0000 7fa3d1209640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3cc14d130 con 0x7fa3cc0b4d00 2024-05-15T12:42:08.798 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.801+0000 7fa3d1209640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:08.798 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.801+0000 7fa3d1209640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3cc14d2b0 con 0x7fa3cc143aa0 2024-05-15T12:42:08.798 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.801+0000 7fa3d1209640 10 monclient(hunting): _renew_subs 2024-05-15T12:42:08.798 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.801+0000 7fa3d1209640 10 monclient(hunting): authenticate will time out at 1285.138640s 2024-05-15T12:42:08.798 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.801+0000 7fa3c3fff640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa3cc0b4d00 0x7fa3cc1d8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:08.798 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.801+0000 7fa3cb59e640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa3cc143aa0 0x7fa3cc1dafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:08.798 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.801+0000 7fa3cad9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa3cc0a94b0 0x7fa3cc143560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:08.799 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.802+0000 7fa3c3fff640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa3cc0b4d00 0x7fa3cc1d8180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.178:44976/0 (socket says 172.21.15.178:44976) 2024-05-15T12:42:08.799 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.802+0000 7fa3c3fff640 1 -- 172.21.15.178:0/3031055157 learned_addr learned my addr 172.21.15.178:0/3031055157 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-05-15T12:42:08.799 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.802+0000 7fa3c3fff640 10 monclient(hunting): get_auth_request con 0x7fa3cc0b4d00 auth_method 0 2024-05-15T12:42:08.799 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.802+0000 7fa3c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:08.799 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.802+0000 7fa3c3fff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:08.799 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.802+0000 7fa3c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:08.799 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.802+0000 7fa3cad9d640 10 monclient(hunting): get_auth_request con 0x7fa3cc0a94b0 auth_method 0 2024-05-15T12:42:08.799 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.802+0000 7fa3cad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:08.799 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.802+0000 7fa3cad9d640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:08.799 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.802+0000 7fa3cad9d640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:08.799 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.802+0000 7fa3c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:08.799 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.802+0000 7fa3c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:08.799 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.802+0000 7fa3c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:08.799 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.802+0000 7fa3cad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:08.799 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.802+0000 7fa3cad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:08.799 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.802+0000 7fa3cad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:08.800 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.803+0000 7fa3cb59e640 1 -- 172.21.15.178:0/3031055157 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa3cc143aa0 msgr2=0x7fa3cc1dafe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-05-15T12:42:08.800 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.803+0000 7fa3cb59e640 1 -- 172.21.15.178:0/3031055157 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa3cc143aa0 msgr2=0x7fa3cc1dafe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-15T12:42:08.800 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.803+0000 7fa3cb59e640 1 --2- 172.21.15.178:0/3031055157 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa3cc143aa0 0x7fa3cc1dafe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-15T12:42:08.800 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.803+0000 7fa3cb59e640 1 --2- 172.21.15.178:0/3031055157 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa3cc143aa0 0x7fa3cc1dafe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-15T12:42:08.800 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.805+0000 7fa3c3fff640 10 monclient(hunting): handle_auth_done global_id 4153 payload 824 2024-05-15T12:42:08.800 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.805+0000 7fa3c3fff640 1 -- 172.21.15.178:0/3031055157 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa3cc143aa0 msgr2=0x7fa3cc1dafe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-15T12:42:08.800 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.805+0000 7fa3c3fff640 1 --2- 172.21.15.178:0/3031055157 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa3cc143aa0 0x7fa3cc1dafe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:08.800 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.805+0000 7fa3c3fff640 1 -- 172.21.15.178:0/3031055157 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa3cc0a94b0 msgr2=0x7fa3cc143560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:08.800 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.805+0000 7fa3c3fff640 1 --2- 172.21.15.178:0/3031055157 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa3cc0a94b0 0x7fa3cc143560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:08.800 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.805+0000 7fa3c3fff640 10 monclient: _finish_hunting 0 2024-05-15T12:42:08.800 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.805+0000 7fa3c3fff640 1 monclient: found mon.noname-b 2024-05-15T12:42:08.800 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.805+0000 7fa3c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:42:08.800 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.805+0000 7fa3c3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2024-05-15T12:42:08.800 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.805+0000 7fa3c3fff640 1 -- 172.21.15.178:0/3031055157 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3cc0ac110 con 0x7fa3cc0b4d00 2024-05-15T12:42:08.801 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.808+0000 7fa3c3fff640 1 --2- 172.21.15.178:0/3031055157 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa3cc0b4d00 0x7fa3cc1d8180 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa3b4009990 tx=0x7fa3b40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:42:08.801 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.808+0000 7fa3c37fe640 10 client.?.objecter ms_handle_connect 0x7fa3cc0b4d00 2024-05-15T12:42:08.801 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.808+0000 7fa3c37fe640 10 client.?.objecter resend_mon_ops 2024-05-15T12:42:08.803 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.812+0000 7fa3c37fe640 1 -- 172.21.15.178:0/3031055157 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3b4040070 con 0x7fa3cc0b4d00 2024-05-15T12:42:08.803 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.812+0000 7fa3c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:08.803 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.812+0000 7fa3c37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:42:08.803 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.812+0000 7fa3c37fe640 10 monclient: dump: 2024-05-15T12:42:08.804 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-05-15T12:42:08.804 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:08.804 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:08.804 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:08.804 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:08.804 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2024-05-15T12:42:08.804 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:08.804 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:08.804 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:08.804 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-05-15T12:42:08.804 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.812+0000 7fa3c37fe640 10 monclient: _finish_auth 0 2024-05-15T12:42:08.804 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.812+0000 7fa3c37fe640 10 monclient: _check_auth_tickets 2024-05-15T12:42:08.804 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.812+0000 7fa3c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:42:08.804 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.812+0000 7fa3c37fe640 1 -- 172.21.15.178:0/3031055157 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3b403e070 con 0x7fa3cc0b4d00 2024-05-15T12:42:08.805 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.812+0000 7fa3d1209640 5 monclient: authenticate success, global_id 4153 2024-05-15T12:42:08.805 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.812+0000 7fa3d1209640 10 monclient: _renew_subs 2024-05-15T12:42:08.805 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.812+0000 7fa3d1209640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:42:08.805 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.812+0000 7fa3d1209640 1 -- 172.21.15.178:0/3031055157 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3cc140500 con 0x7fa3cc0b4d00 2024-05-15T12:42:08.805 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.812+0000 7fa3c37fe640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:42:08.805 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.812+0000 7fa3c37fe640 1 -- 172.21.15.178:0/3031055157 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3b4035040 con 0x7fa3cc0b4d00 2024-05-15T12:42:08.805 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.812+0000 7fa3c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:08.805 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.812+0000 7fa3c37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:42:08.805 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.812+0000 7fa3c37fe640 10 monclient: dump: 2024-05-15T12:42:08.805 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-05-15T12:42:08.805 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:08.805 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:08.805 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:08.806 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:08.806 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2024-05-15T12:42:08.806 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:08.806 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:08.806 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:08.806 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-05-15T12:42:08.806 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.812+0000 7fa3c9d9b640 4 set_mon_vals no callback set 2024-05-15T12:42:08.806 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.812+0000 7fa3d1209640 10 client.4153.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:42:08.806 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.812+0000 7fa3d1209640 10 monclient: _renew_subs 2024-05-15T12:42:08.806 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.812+0000 7fa3d1209640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:42:08.806 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.812+0000 7fa3d1209640 1 -- 172.21.15.178:0/3031055157 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3cc1d88a0 con 0x7fa3cc0b4d00 2024-05-15T12:42:08.806 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.812+0000 7fa3d1209640 1 librados: init done 2024-05-15T12:42:08.806 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.813+0000 7fa3d1209640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:42:08.806 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.813+0000 7fa3d1209640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:42:08.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.813+0000 7fa3d1209640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:42:08.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.813+0000 7fa3d1209640 1 -- 172.21.15.178:0/3031055157 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3cc04f0e0 con 0x7fa3cc0b4d00 2024-05-15T12:42:08.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.815+0000 7fa3c37fe640 1 -- 172.21.15.178:0/3031055157 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fa3b4038020 con 0x7fa3cc0b4d00 2024-05-15T12:42:08.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.815+0000 7fa3c37fe640 1 -- 172.21.15.178:0/3031055157 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7fa3b403a9a0 con 0x7fa3cc0b4d00 2024-05-15T12:42:08.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.815+0000 7fa3c37fe640 10 client.4153.objecter ms_dispatch 0x7fa3cc000b90 osd_map(7..7 src has 1..7) v4 2024-05-15T12:42:08.808 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.815+0000 7fa3c37fe640 3 client.4153.objecter handle_osd_map got epochs [7,7] > 0 2024-05-15T12:42:08.808 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.815+0000 7fa3c37fe640 3 client.4153.objecter handle_osd_map decoding full epoch 7 2024-05-15T12:42:08.808 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.815+0000 7fa3c37fe640 20 client.4153.objecter dump_active .. 0 homeless 2024-05-15T12:42:08.808 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-05-15T12:42:08.818+0000 7f5134c38040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-15T12:42:08.825 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-05-15T12:42:08.835+0000 7f18296bb040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-15T12:42:08.878 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-05-15T12:42:08.888+0000 7f5134c38040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-15T12:42:08.904 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.914+0000 7fa3c37fe640 1 -- 172.21.15.178:0/3031055157 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fa3cc0b16d0 con 0x7fa3cc0b4d00 2024-05-15T12:42:08.904 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.914+0000 7fa3c37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:42:08.904 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.914+0000 7fa3c37fe640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:42:08.923 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-05-15T12:42:08.933+0000 7f5134c38040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-15T12:42:08.966 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-05-15T12:42:08.975+0000 7f18296bb040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-15T12:42:08.977 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.987+0000 7fa3d1209640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "c5f9bc37-4bd6-4643-8f0d-e53c5f2c8782", "id": 5}] 2024-05-15T12:42:08.977 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.987+0000 7fa3d1209640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "c5f9bc37-4bd6-4643-8f0d-e53c5f2c8782", "id": 5}] 2024-05-15T12:42:08.977 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.987+0000 7fa3d1209640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:42:08.977 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:08.987+0000 7fa3d1209640 1 -- 172.21.15.178:0/3031055157 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "osd new", "uuid": "c5f9bc37-4bd6-4643-8f0d-e53c5f2c8782", "id": 5} v 0) v1 -- 0x7fa3cc067d90 con 0x7fa3cc0b4d00 2024-05-15T12:42:09.017 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-05-15T12:42:09.027+0000 7f18296bb040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-15T12:42:09.065 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.075+0000 7fa3c37fe640 1 -- 172.21.15.178:0/3031055157 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c5f9bc37-4bd6-4643-8f0d-e53c5f2c8782", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7fa3b4045460 con 0x7fa3cc0b4d00 2024-05-15T12:42:09.066 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.075+0000 7fa3c37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "c5f9bc37-4bd6-4643-8f0d-e53c5f2c8782", "id": 5}] 2024-05-15T12:42:09.066 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.075+0000 7fa3c37fe640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:42:09.066 INFO:teuthology.orchestra.run.smithi178.stdout:5 2024-05-15T12:42:09.068 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.076+0000 7fa3d1209640 10 librados: watch_flush enter 2024-05-15T12:42:09.068 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.076+0000 7fa3d1209640 10 librados: watch_flush exit 2024-05-15T12:42:09.068 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.076+0000 7fa3d1209640 20 client.4153.objecter shutdown clearing up homeless session... 2024-05-15T12:42:09.068 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.076+0000 7fa3d1209640 10 client.4153.objecter successfully canceled tick 2024-05-15T12:42:09.068 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.076+0000 7fa3d1209640 10 monclient: shutdown 2024-05-15T12:42:09.068 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.076+0000 7fa3d1209640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:42:09.068 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.076+0000 7fa3d1209640 1 -- 172.21.15.178:0/3031055157 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa3cc0b4d00 msgr2=0x7fa3cc1d8180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:09.068 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.076+0000 7fa3d1209640 1 --2- 172.21.15.178:0/3031055157 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa3cc0b4d00 0x7fa3cc1d8180 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa3b4009990 tx=0x7fa3b40046c0 comp rx=0 tx=0).stop 2024-05-15T12:42:09.068 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.076+0000 7fa3d1209640 1 -- 172.21.15.178:0/3031055157 shutdown_connections 2024-05-15T12:42:09.068 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.076+0000 7fa3d1209640 1 --2- 172.21.15.178:0/3031055157 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa3cc143aa0 0x7fa3cc1dafe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:09.068 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.076+0000 7fa3d1209640 1 --2- 172.21.15.178:0/3031055157 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa3cc0b4d00 0x7fa3cc1d8180 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:09.069 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.076+0000 7fa3d1209640 1 --2- 172.21.15.178:0/3031055157 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa3cc0a94b0 0x7fa3cc143560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:09.069 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.076+0000 7fa3d1209640 1 -- 172.21.15.178:0/3031055157 >> 172.21.15.178:0/3031055157 conn(0x7fa3cc066cc0 msgr2=0x7fa3cc0b5b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:42:09.069 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.076+0000 7fa3d1209640 1 -- 172.21.15.178:0/3031055157 shutdown_connections 2024-05-15T12:42:09.069 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.076+0000 7fa3d1209640 1 -- 172.21.15.178:0/3031055157 wait complete. 2024-05-15T12:42:09.069 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.077+0000 7fa3d1209640 1 librados: shutdown 2024-05-15T12:42:09.078 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph --cluster ceph osd new d5af075d-3eaa-4cb5-b6e9-6f01c201f2ba 6 2024-05-15T12:42:09.092 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-05-15T12:42:09.102+0000 7f18296bb040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-15T12:42:09.143 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-05-15T12:42:09.153+0000 7f18296bb040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-15T12:42:09.179 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.188+0000 7f09d76b7640 10 monclient: get_monmap_and_config 2024-05-15T12:42:09.179 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.188+0000 7f09d76b7640 10 monclient: build_initial_monmap 2024-05-15T12:42:09.179 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.188+0000 7f09d76b7640 10 monclient: monmap: 2024-05-15T12:42:09.179 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 0 2024-05-15T12:42:09.179 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:09.179 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-05-15T12:42:09.189671+0000 2024-05-15T12:42:09.179 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-05-15T12:42:09.189671+0000 2024-05-15T12:42:09.180 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 0 (unknown) 2024-05-15T12:42:09.180 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2024-05-15T12:42:09.180 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T12:42:09.180 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T12:42:09.180 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T12:42:09.180 INFO:teuthology.orchestra.run.smithi178.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T12:42:09.180 INFO:teuthology.orchestra.run.smithi178.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T12:42:09.180 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-05-15T12:42:09.180 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.189+0000 7f09d76b7640 1 Processor -- start 2024-05-15T12:42:09.180 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.189+0000 7f09d76b7640 1 -- start start 2024-05-15T12:42:09.180 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.189+0000 7f09d76b7640 10 monclient: init 2024-05-15T12:42:09.183 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.190+0000 7f09d76b7640 10 monclient: _reopen_session rank -1 2024-05-15T12:42:09.183 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.190+0000 7f09d76b7640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-05-15T12:42:09.183 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.190+0000 7f09d76b7640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f09d00b4d00 0x7f09d00b1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:09.184 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.190+0000 7f09d76b7640 10 monclient(hunting): picked mon.noname-c con 0x7f09d00b4d00 addr v2:172.21.15.178:3300/0 2024-05-15T12:42:09.184 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.190+0000 7f09d76b7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f09d00b50e0 addr v1:172.21.15.117:6789/0 2024-05-15T12:42:09.184 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.190+0000 7f09d76b7640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f09d00a94b0 0x7f09d00b0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:09.184 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.190+0000 7f09d76b7640 10 monclient(hunting): picked mon.noname-b con 0x7f09d00a94b0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:42:09.184 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.190+0000 7f09d76b7640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09d00ab1b0 con 0x7f09d00b50e0 2024-05-15T12:42:09.184 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.190+0000 7f09d76b7640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:09.184 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.190+0000 7f09d76b7640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09d00b23c0 con 0x7f09d00b4d00 2024-05-15T12:42:09.184 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.190+0000 7f09d76b7640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:09.184 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.190+0000 7f09d76b7640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09d00b1520 con 0x7f09d00a94b0 2024-05-15T12:42:09.184 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.190+0000 7f09d76b7640 10 monclient(hunting): _renew_subs 2024-05-15T12:42:09.184 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.190+0000 7f09d76b7640 10 monclient(hunting): authenticate will time out at 1285.527566s 2024-05-15T12:42:09.184 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.190+0000 7f09d5453640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f09d00b4d00 0x7f09d00b1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:09.184 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.190+0000 7f09d4c52640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f09d00b50e0 0x7f09d0145c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.178:49784/0 (socket says 172.21.15.178:49784) 2024-05-15T12:42:09.184 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.190+0000 7f09d4c52640 1 -- 172.21.15.178:0/808839467 learned_addr learned my addr 172.21.15.178:0/808839467 (peer_addr_for_me v1:172.21.15.178:0/0) 2024-05-15T12:42:09.184 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.190+0000 7f09d5c54640 1 --2- 172.21.15.178:0/808839467 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f09d00a94b0 0x7f09d00b0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:09.185 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.191+0000 7f09d5453640 10 monclient(hunting): get_auth_request con 0x7f09d00b4d00 auth_method 0 2024-05-15T12:42:09.185 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.191+0000 7f09d5453640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:09.185 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.191+0000 7f09d5453640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:09.185 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.191+0000 7f09d5453640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:09.185 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.191+0000 7f09d5c54640 10 monclient(hunting): get_auth_request con 0x7f09d00a94b0 auth_method 0 2024-05-15T12:42:09.185 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.191+0000 7f09d5c54640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:09.185 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.191+0000 7f09d5c54640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:09.185 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.191+0000 7f09d5c54640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:09.185 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.191+0000 7f09d5453640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:09.185 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.191+0000 7f09d5453640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:09.185 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.191+0000 7f09d5453640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:09.185 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.191+0000 7f09d5c54640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:09.185 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.191+0000 7f09d5c54640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:09.185 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.191+0000 7f09d5c54640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:09.186 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.193+0000 7f09d5453640 10 monclient(hunting): handle_auth_done global_id 4156 payload 274 2024-05-15T12:42:09.186 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.193+0000 7f09d5453640 1 -- 172.21.15.178:0/808839467 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f09d00a94b0 msgr2=0x7f09d00b0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:42:09.186 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.193+0000 7f09d5453640 1 --2- 172.21.15.178:0/808839467 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f09d00a94b0 0x7f09d00b0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:09.186 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.193+0000 7f09d5453640 1 -- 172.21.15.178:0/808839467 >> v1:172.21.15.117:6789/0 conn(0x7f09d00b50e0 legacy=0x7f09d0145c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:09.186 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.193+0000 7f09d5453640 10 monclient: _finish_hunting 0 2024-05-15T12:42:09.186 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.193+0000 7f09d5453640 1 monclient: found mon.noname-c 2024-05-15T12:42:09.186 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.193+0000 7f09c7fff640 1 -- 172.21.15.178:0/808839467 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2472642692 0 0) 0x7f09d00ab1b0 con 0x7f09d00b50e0 2024-05-15T12:42:09.186 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.193+0000 7f09d5453640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:42:09.186 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.193+0000 7f09d5453640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.178:3300/0 2024-05-15T12:42:09.186 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.193+0000 7f09d5453640 1 -- 172.21.15.178:0/808839467 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d00b16d0 con 0x7f09d00b4d00 2024-05-15T12:42:09.186 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.193+0000 7f09c7fff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-05-15T12:42:09.187 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.195+0000 7f09d5453640 1 --2- 172.21.15.178:0/808839467 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f09d00b4d00 0x7f09d00b1e80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f09cc0097a0 tx=0x7f09cc02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0204834ae4a5dc server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:42:09.189 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.199+0000 7f09c7fff640 1 -- 172.21.15.178:0/808839467 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09cc0315a0 con 0x7f09d00b4d00 2024-05-15T12:42:09.190 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.199+0000 7f09c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:09.190 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.199+0000 7f09c7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:42:09.190 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.199+0000 7f09c7fff640 10 monclient: dump: 2024-05-15T12:42:09.190 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-05-15T12:42:09.190 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:09.190 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:09.190 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:09.190 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:09.190 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2024-05-15T12:42:09.190 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:09.190 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:09.190 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:09.191 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-05-15T12:42:09.191 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.199+0000 7f09c7fff640 10 monclient: _finish_auth 0 2024-05-15T12:42:09.191 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.199+0000 7f09c7fff640 10 monclient: _check_auth_tickets 2024-05-15T12:42:09.191 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.199+0000 7f09c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:42:09.191 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.199+0000 7f09c7fff640 1 -- 172.21.15.178:0/808839467 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09cc037070 con 0x7f09d00b4d00 2024-05-15T12:42:09.191 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.199+0000 7f09c7fff640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:42:09.191 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.199+0000 7f09c7fff640 1 -- 172.21.15.178:0/808839467 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09cc031960 con 0x7f09d00b4d00 2024-05-15T12:42:09.191 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.199+0000 7f09d76b7640 5 monclient: authenticate success, global_id 4156 2024-05-15T12:42:09.191 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.199+0000 7f09d76b7640 10 monclient: get_monmap_and_config success 2024-05-15T12:42:09.191 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.199+0000 7f09d76b7640 4 set_mon_vals no callback set 2024-05-15T12:42:09.191 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.199+0000 7f09c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:09.191 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.199+0000 7f09c7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:42:09.191 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.199+0000 7f09c7fff640 10 monclient: dump: 2024-05-15T12:42:09.192 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-05-15T12:42:09.192 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:09.192 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:09.192 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:09.192 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:09.192 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2024-05-15T12:42:09.192 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:09.192 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:09.192 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:09.192 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-05-15T12:42:09.196 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.200+0000 7f09d76b7640 10 monclient: shutdown 2024-05-15T12:42:09.196 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.200+0000 7f09d76b7640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:42:09.196 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.200+0000 7f09d76b7640 1 -- 172.21.15.178:0/808839467 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f09d00b4d00 msgr2=0x7f09d00b1e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:09.196 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.200+0000 7f09d76b7640 1 --2- 172.21.15.178:0/808839467 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f09d00b4d00 0x7f09d00b1e80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f09cc0097a0 tx=0x7f09cc02b840 comp rx=0 tx=0).stop 2024-05-15T12:42:09.196 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.200+0000 7f09d76b7640 1 -- 172.21.15.178:0/808839467 shutdown_connections 2024-05-15T12:42:09.196 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.200+0000 7f09d76b7640 1 --2- 172.21.15.178:0/808839467 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f09d00a94b0 0x7f09d00b0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:09.196 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.200+0000 7f09d76b7640 1 --2- 172.21.15.178:0/808839467 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f09d00b4d00 0x7f09d00b1e80 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:09.197 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.200+0000 7f09d76b7640 1 -- 172.21.15.178:0/808839467 >> 172.21.15.178:0/808839467 conn(0x7f09d0066cc0 msgr2=0x7f09d00670d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:42:09.197 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.201+0000 7f09d76b7640 1 -- 172.21.15.178:0/808839467 shutdown_connections 2024-05-15T12:42:09.197 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.201+0000 7f09d76b7640 1 -- 172.21.15.178:0/808839467 wait complete. 2024-05-15T12:42:09.197 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.201+0000 7f09d76b7640 10 monclient: build_initial_monmap 2024-05-15T12:42:09.197 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.201+0000 7f09d76b7640 10 monclient: monmap: 2024-05-15T12:42:09.197 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 0 2024-05-15T12:42:09.197 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:42:09.197 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 0.000000 2024-05-15T12:42:09.197 INFO:teuthology.orchestra.run.smithi178.stderr:created 0.000000 2024-05-15T12:42:09.197 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 0 (unknown) 2024-05-15T12:42:09.197 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2024-05-15T12:42:09.197 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T12:42:09.197 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T12:42:09.197 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T12:42:09.197 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-05-15T12:42:09.198 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.202+0000 7f09d76b7640 1 librados: starting msgr at 2024-05-15T12:42:09.198 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.202+0000 7f09d76b7640 1 librados: starting objecter 2024-05-15T12:42:09.198 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.202+0000 7f09d76b7640 1 Processor -- start 2024-05-15T12:42:09.198 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.202+0000 7f09d76b7640 1 -- start start 2024-05-15T12:42:09.198 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.202+0000 7f09d76b7640 1 librados: setting wanted keys 2024-05-15T12:42:09.198 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.202+0000 7f09d76b7640 1 librados: calling monclient init 2024-05-15T12:42:09.198 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.202+0000 7f09d76b7640 10 monclient: init 2024-05-15T12:42:09.198 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.202+0000 7f09d76b7640 10 monclient: _reopen_session rank -1 2024-05-15T12:42:09.198 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.202+0000 7f09d76b7640 10 monclient: _add_conns ranks=[2,1,0] 2024-05-15T12:42:09.198 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.202+0000 7f09d76b7640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f09d00a94b0 0x7f09d0142370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:09.198 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.202+0000 7f09d76b7640 10 monclient(hunting): picked mon.noname-c con 0x7f09d00a94b0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:42:09.198 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.202+0000 7f09d76b7640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f09d00b4d00 0x7f09d01dadd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:09.198 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.202+0000 7f09d76b7640 10 monclient(hunting): picked mon.noname-b con 0x7f09d00b4d00 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:42:09.198 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.202+0000 7f09d76b7640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f09d01db310 0x7f09d013b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:09.199 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.202+0000 7f09d76b7640 10 monclient(hunting): picked mon.noname-a con 0x7f09d01db310 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T12:42:09.199 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.202+0000 7f09d76b7640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:09.199 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.202+0000 7f09d76b7640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d014d130 con 0x7f09d01db310 2024-05-15T12:42:09.199 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.202+0000 7f09d76b7640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:09.199 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.202+0000 7f09d76b7640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d014cfb0 con 0x7f09d00b4d00 2024-05-15T12:42:09.199 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.202+0000 7f09d76b7640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:09.199 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.202+0000 7f09d76b7640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09d014d2b0 con 0x7f09d00a94b0 2024-05-15T12:42:09.199 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.202+0000 7f09d76b7640 10 monclient(hunting): _renew_subs 2024-05-15T12:42:09.199 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.202+0000 7f09d76b7640 10 monclient(hunting): authenticate will time out at 1285.540278s 2024-05-15T12:42:09.199 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.203+0000 7f09d4c52640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f09d00b4d00 0x7f09d01dadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:09.199 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.203+0000 7f09d5c54640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f09d01db310 0x7f09d013b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:09.199 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.203+0000 7f09d4c52640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f09d00b4d00 0x7f09d01dadd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.178:45002/0 (socket says 172.21.15.178:45002) 2024-05-15T12:42:09.200 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.203+0000 7f09d4c52640 1 -- 172.21.15.178:0/1669360790 learned_addr learned my addr 172.21.15.178:0/1669360790 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-05-15T12:42:09.200 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.203+0000 7f09d4c52640 10 monclient(hunting): get_auth_request con 0x7f09d00b4d00 auth_method 0 2024-05-15T12:42:09.200 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.203+0000 7f09d4c52640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:09.200 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.203+0000 7f09d4c52640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:09.200 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.203+0000 7f09d4c52640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:09.200 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.203+0000 7f09d5453640 1 --2- 172.21.15.178:0/1669360790 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f09d00a94b0 0x7f09d0142370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:09.200 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.203+0000 7f09d5c54640 10 monclient(hunting): get_auth_request con 0x7f09d01db310 auth_method 0 2024-05-15T12:42:09.200 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.203+0000 7f09d5c54640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:09.200 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.203+0000 7f09d5c54640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:09.200 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.203+0000 7f09d5c54640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:09.200 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.204+0000 7f09d5453640 10 monclient(hunting): get_auth_request con 0x7f09d00a94b0 auth_method 0 2024-05-15T12:42:09.200 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.204+0000 7f09d5453640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:09.200 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.204+0000 7f09d5453640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:09.200 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.204+0000 7f09d5453640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:09.201 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.204+0000 7f09d4c52640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:09.201 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.204+0000 7f09d4c52640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:09.201 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.204+0000 7f09d4c52640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:09.201 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.204+0000 7f09d5c54640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:09.201 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.204+0000 7f09d5c54640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:09.201 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.204+0000 7f09d5c54640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:09.201 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.204+0000 7f09d5453640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:09.201 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.204+0000 7f09d5453640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:09.201 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.204+0000 7f09d5453640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:09.202 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.207+0000 7f09d5c54640 10 monclient(hunting): handle_auth_done global_id 4152 payload 824 2024-05-15T12:42:09.202 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.207+0000 7f09d5c54640 1 -- 172.21.15.178:0/1669360790 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f09d00a94b0 msgr2=0x7f09d0142370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:09.202 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.207+0000 7f09d5c54640 1 --2- 172.21.15.178:0/1669360790 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f09d00a94b0 0x7f09d0142370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:09.202 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.207+0000 7f09d5c54640 1 -- 172.21.15.178:0/1669360790 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f09d00b4d00 msgr2=0x7f09d01dadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:09.203 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.207+0000 7f09d5c54640 1 --2- 172.21.15.178:0/1669360790 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f09d00b4d00 0x7f09d01dadd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:09.203 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.207+0000 7f09d5c54640 10 monclient: _finish_hunting 0 2024-05-15T12:42:09.203 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.207+0000 7f09d5c54640 1 monclient: found mon.noname-a 2024-05-15T12:42:09.203 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.207+0000 7f09d5c54640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:42:09.203 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.207+0000 7f09d5c54640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.117:3300/0 2024-05-15T12:42:09.203 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.207+0000 7f09d5c54640 1 -- 172.21.15.178:0/1669360790 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d01db8e0 con 0x7f09d01db310 2024-05-15T12:42:09.203 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.209+0000 7f09d5c54640 1 --2- 172.21.15.178:0/1669360790 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f09d01db310 0x7f09d013b140 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f09c800efc0 tx=0x7f09c800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:42:09.203 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.209+0000 7f09c5ffb640 10 client.?.objecter ms_handle_connect 0x7f09d01db310 2024-05-15T12:42:09.203 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.209+0000 7f09c5ffb640 10 client.?.objecter resend_mon_ops 2024-05-15T12:42:09.204 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.213+0000 7f09c5ffb640 1 -- 172.21.15.178:0/1669360790 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c8018020 con 0x7f09d01db310 2024-05-15T12:42:09.205 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.213+0000 7f09c5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:09.205 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.213+0000 7f09c5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T12:42:09.205 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.213+0000 7f09c5ffb640 10 monclient: dump: 2024-05-15T12:42:09.205 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-05-15T12:42:09.205 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:09.205 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:09.205 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:09.205 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:09.205 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2024-05-15T12:42:09.205 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:09.205 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:09.205 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:09.205 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-05-15T12:42:09.205 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.213+0000 7f09c5ffb640 10 monclient: _finish_auth 0 2024-05-15T12:42:09.205 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.213+0000 7f09c5ffb640 10 monclient: _check_auth_tickets 2024-05-15T12:42:09.205 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.213+0000 7f09c5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:42:09.206 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.213+0000 7f09c5ffb640 1 -- 172.21.15.178:0/1669360790 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09c8014480 con 0x7f09d01db310 2024-05-15T12:42:09.206 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.213+0000 7f09d76b7640 5 monclient: authenticate success, global_id 4152 2024-05-15T12:42:09.206 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.213+0000 7f09c5ffb640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:42:09.206 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.213+0000 7f09c5ffb640 1 -- 172.21.15.178:0/1669360790 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c8009b10 con 0x7f09d01db310 2024-05-15T12:42:09.206 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.213+0000 7f09c5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:09.206 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.213+0000 7f09c5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T12:42:09.206 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.213+0000 7f09c5ffb640 10 monclient: dump: 2024-05-15T12:42:09.206 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-05-15T12:42:09.206 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:09.206 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:09.206 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:09.206 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:09.206 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2024-05-15T12:42:09.206 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:09.207 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:09.207 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:09.207 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-05-15T12:42:09.207 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.213+0000 7f09d76b7640 10 monclient: _renew_subs 2024-05-15T12:42:09.207 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.213+0000 7f09d76b7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:42:09.207 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.213+0000 7f09d76b7640 1 -- 172.21.15.178:0/1669360790 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09d013b680 con 0x7f09d01db310 2024-05-15T12:42:09.207 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.213+0000 7f09c77fe640 4 set_mon_vals no callback set 2024-05-15T12:42:09.207 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.213+0000 7f09d76b7640 10 client.4152.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:42:09.207 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.213+0000 7f09d76b7640 10 monclient: _renew_subs 2024-05-15T12:42:09.207 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.213+0000 7f09d76b7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:42:09.207 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.213+0000 7f09d76b7640 1 -- 172.21.15.178:0/1669360790 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09d013bb80 con 0x7f09d01db310 2024-05-15T12:42:09.207 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.213+0000 7f09d76b7640 1 librados: init done 2024-05-15T12:42:09.207 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.214+0000 7f09d76b7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:42:09.207 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.214+0000 7f09d76b7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:42:09.207 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.214+0000 7f09d76b7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:42:09.207 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.214+0000 7f09d76b7640 1 -- 172.21.15.178:0/1669360790 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09d004f0e0 con 0x7f09d01db310 2024-05-15T12:42:09.300 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.310+0000 7f09c5ffb640 1 -- 172.21.15.178:0/1669360790 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f09c8005d10 con 0x7f09d01db310 2024-05-15T12:42:09.305 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.315+0000 7f09c5ffb640 1 -- 172.21.15.178:0/1669360790 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f09c801dde0 con 0x7f09d01db310 2024-05-15T12:42:09.305 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.315+0000 7f09c5ffb640 10 client.4152.objecter ms_dispatch 0x7f09d0000b90 osd_map(8..8 src has 1..8) v4 2024-05-15T12:42:09.305 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.315+0000 7f09c5ffb640 3 client.4152.objecter handle_osd_map got epochs [8,8] > 0 2024-05-15T12:42:09.306 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.315+0000 7f09c5ffb640 3 client.4152.objecter handle_osd_map decoding full epoch 8 2024-05-15T12:42:09.306 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.315+0000 7f09c5ffb640 20 client.4152.objecter dump_active .. 0 homeless 2024-05-15T12:42:09.306 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.315+0000 7f09c5ffb640 1 -- 172.21.15.178:0/1669360790 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f09c8022030 con 0x7f09d01db310 2024-05-15T12:42:09.306 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.315+0000 7f09c5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:42:09.306 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.315+0000 7f09c5ffb640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:42:09.379 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.388+0000 7f09d76b7640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d5af075d-3eaa-4cb5-b6e9-6f01c201f2ba", "id": 6}] 2024-05-15T12:42:09.379 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.388+0000 7f09d76b7640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d5af075d-3eaa-4cb5-b6e9-6f01c201f2ba", "id": 6}] 2024-05-15T12:42:09.379 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.388+0000 7f09d76b7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:42:09.379 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.388+0000 7f09d76b7640 1 -- 172.21.15.178:0/1669360790 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd new", "uuid": "d5af075d-3eaa-4cb5-b6e9-6f01c201f2ba", "id": 6} v 0) v1 -- 0x7f09d00b74e0 con 0x7f09d01db310 2024-05-15T12:42:09.408 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.418+0000 7f09c5ffb640 1 -- 172.21.15.178:0/1669360790 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d5af075d-3eaa-4cb5-b6e9-6f01c201f2ba", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f09c8010040 con 0x7f09d01db310 2024-05-15T12:42:09.408 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.418+0000 7f09c5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d5af075d-3eaa-4cb5-b6e9-6f01c201f2ba", "id": 6}] 2024-05-15T12:42:09.408 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.418+0000 7f09c5ffb640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:42:09.409 INFO:teuthology.orchestra.run.smithi178.stdout:6 2024-05-15T12:42:09.410 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.419+0000 7f09d76b7640 10 librados: watch_flush enter 2024-05-15T12:42:09.410 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.419+0000 7f09d76b7640 10 librados: watch_flush exit 2024-05-15T12:42:09.410 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.419+0000 7f09d76b7640 20 client.4152.objecter shutdown clearing up homeless session... 2024-05-15T12:42:09.410 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.419+0000 7f09d76b7640 10 client.4152.objecter successfully canceled tick 2024-05-15T12:42:09.410 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.419+0000 7f09d76b7640 10 monclient: shutdown 2024-05-15T12:42:09.410 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.419+0000 7f09d76b7640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:42:09.410 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.419+0000 7f09d76b7640 1 -- 172.21.15.178:0/1669360790 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f09d01db310 msgr2=0x7f09d013b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:09.411 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.419+0000 7f09d76b7640 1 --2- 172.21.15.178:0/1669360790 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f09d01db310 0x7f09d013b140 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f09c800efc0 tx=0x7f09c800c490 comp rx=0 tx=0).stop 2024-05-15T12:42:09.411 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.419+0000 7f09d76b7640 1 -- 172.21.15.178:0/1669360790 shutdown_connections 2024-05-15T12:42:09.411 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.419+0000 7f09d76b7640 1 --2- 172.21.15.178:0/1669360790 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f09d01db310 0x7f09d013b140 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:09.411 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.419+0000 7f09d76b7640 1 --2- 172.21.15.178:0/1669360790 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f09d00b4d00 0x7f09d01dadd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:09.411 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.419+0000 7f09d76b7640 1 --2- 172.21.15.178:0/1669360790 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f09d00a94b0 0x7f09d0142370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:09.411 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.419+0000 7f09d76b7640 1 -- 172.21.15.178:0/1669360790 >> 172.21.15.178:0/1669360790 conn(0x7f09d0066cc0 msgr2=0x7f09d00b6600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:42:09.411 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.419+0000 7f09d76b7640 1 -- 172.21.15.178:0/1669360790 shutdown_connections 2024-05-15T12:42:09.411 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.419+0000 7f09d76b7640 1 -- 172.21.15.178:0/1669360790 wait complete. 2024-05-15T12:42:09.411 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.419+0000 7f09d76b7640 1 librados: shutdown 2024-05-15T12:42:09.423 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph --cluster ceph osd new 323233c7-a3af-4fb0-ac70-de523a279b44 7 2024-05-15T12:42:09.466 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-05-15T12:42:09.477+0000 7f5134c38040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-15T12:42:09.563 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.573+0000 7fb47953a640 10 monclient: get_monmap_and_config 2024-05-15T12:42:09.563 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.573+0000 7fb47953a640 10 monclient: build_initial_monmap 2024-05-15T12:42:09.564 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.573+0000 7fb47953a640 10 monclient: monmap: 2024-05-15T12:42:09.564 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 0 2024-05-15T12:42:09.564 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:09.564 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-05-15T12:42:09.574591+0000 2024-05-15T12:42:09.564 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-05-15T12:42:09.574591+0000 2024-05-15T12:42:09.564 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 0 (unknown) 2024-05-15T12:42:09.564 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2024-05-15T12:42:09.564 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T12:42:09.564 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T12:42:09.564 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T12:42:09.564 INFO:teuthology.orchestra.run.smithi178.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T12:42:09.564 INFO:teuthology.orchestra.run.smithi178.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T12:42:09.564 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-05-15T12:42:09.567 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.574+0000 7fb47953a640 1 Processor -- start 2024-05-15T12:42:09.567 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.574+0000 7fb47953a640 1 -- start start 2024-05-15T12:42:09.567 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.574+0000 7fb47953a640 10 monclient: init 2024-05-15T12:42:09.567 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.574+0000 7fb47953a640 10 monclient: _reopen_session rank -1 2024-05-15T12:42:09.567 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.575+0000 7fb47953a640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-05-15T12:42:09.567 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.575+0000 7fb47953a640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fb474070aa0 0x7fb474140810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:09.567 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.575+0000 7fb47953a640 10 monclient(hunting): picked mon.noname-c con 0x7fb474070aa0 addr v2:172.21.15.178:3300/0 2024-05-15T12:42:09.567 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.575+0000 7fb47953a640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb474140d50 0x7fb4741439d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:09.568 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.575+0000 7fb47953a640 10 monclient(hunting): picked mon.noname-b con 0x7fb474140d50 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:42:09.568 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.575+0000 7fb47953a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb474143f10 addr v1:172.21.15.117:6789/0 2024-05-15T12:42:09.568 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.575+0000 7fb47953a640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb474072730 con 0x7fb474143f10 2024-05-15T12:42:09.568 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.575+0000 7fb47953a640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:09.568 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.575+0000 7fb47953a640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4740b06e0 con 0x7fb474070aa0 2024-05-15T12:42:09.568 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.575+0000 7fb47953a640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:09.568 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.575+0000 7fb47953a640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb474072a10 con 0x7fb474140d50 2024-05-15T12:42:09.568 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.575+0000 7fb47953a640 10 monclient(hunting): _renew_subs 2024-05-15T12:42:09.568 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.575+0000 7fb47953a640 10 monclient(hunting): authenticate will time out at 1285.912504s 2024-05-15T12:42:09.568 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.575+0000 7fb472ffd640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fb474070aa0 0x7fb474140810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:09.568 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.575+0000 7fb4737fe640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fb474143f10 0x7fb47413cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.178:49788/0 (socket says 172.21.15.178:49788) 2024-05-15T12:42:09.568 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.575+0000 7fb4737fe640 1 -- 172.21.15.178:0/768645901 learned_addr learned my addr 172.21.15.178:0/768645901 (peer_addr_for_me v1:172.21.15.178:0/0) 2024-05-15T12:42:09.568 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.575+0000 7fb46bfff640 1 --2- 172.21.15.178:0/768645901 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb474140d50 0x7fb4741439d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:09.568 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.576+0000 7fb472ffd640 10 monclient(hunting): get_auth_request con 0x7fb474070aa0 auth_method 0 2024-05-15T12:42:09.569 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.576+0000 7fb472ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:09.569 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.576+0000 7fb472ffd640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:09.569 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.576+0000 7fb472ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:09.569 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.576+0000 7fb46bfff640 10 monclient(hunting): get_auth_request con 0x7fb474140d50 auth_method 0 2024-05-15T12:42:09.569 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.576+0000 7fb46bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:09.569 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.576+0000 7fb46bfff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:09.569 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.576+0000 7fb46bfff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:09.569 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.576+0000 7fb472ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:09.569 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.576+0000 7fb472ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:09.569 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.576+0000 7fb472ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:09.569 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.576+0000 7fb46bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:09.569 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.576+0000 7fb46bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:09.569 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.576+0000 7fb46bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:09.570 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.578+0000 7fb4727fc640 1 -- 172.21.15.178:0/768645901 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3705886889 0 0) 0x7fb474072730 con 0x7fb474143f10 2024-05-15T12:42:09.571 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.578+0000 7fb4727fc640 10 monclient(hunting): _negotiate 2024-05-15T12:42:09.571 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.578+0000 7fb4727fc640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:09.571 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.578+0000 7fb4727fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:09.571 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.578+0000 7fb4727fc640 10 monclient(hunting): my global_id is 4155 2024-05-15T12:42:09.571 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.578+0000 7fb4727fc640 1 -- 172.21.15.178:0/768645901 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb460003520 con 0x7fb474143f10 2024-05-15T12:42:09.571 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.578+0000 7fb472ffd640 10 monclient(hunting): handle_auth_done global_id 4162 payload 274 2024-05-15T12:42:09.571 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.578+0000 7fb472ffd640 1 -- 172.21.15.178:0/768645901 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb474140d50 msgr2=0x7fb4741439d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:42:09.571 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.578+0000 7fb472ffd640 1 --2- 172.21.15.178:0/768645901 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb474140d50 0x7fb4741439d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:09.571 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.578+0000 7fb472ffd640 1 -- 172.21.15.178:0/768645901 >> v1:172.21.15.117:6789/0 conn(0x7fb474143f10 legacy=0x7fb47413cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:09.571 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.578+0000 7fb472ffd640 10 monclient: _finish_hunting 0 2024-05-15T12:42:09.571 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.578+0000 7fb472ffd640 1 monclient: found mon.noname-c 2024-05-15T12:42:09.571 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.578+0000 7fb472ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:42:09.571 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.578+0000 7fb472ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.178:3300/0 2024-05-15T12:42:09.571 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.578+0000 7fb472ffd640 1 -- 172.21.15.178:0/768645901 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb474072cf0 con 0x7fb474070aa0 2024-05-15T12:42:09.571 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.580+0000 7fb472ffd640 1 --2- 172.21.15.178:0/768645901 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb474070aa0 0x7fb474140810 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb4640097a0 tx=0x7fb46402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=73473d66c7053344 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:42:09.574 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.582+0000 7fb4727fc640 1 -- 172.21.15.178:0/768645901 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4640315a0 con 0x7fb474070aa0 2024-05-15T12:42:09.574 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.582+0000 7fb4727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:09.574 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.582+0000 7fb4727fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:42:09.575 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.582+0000 7fb4727fc640 10 monclient: dump: 2024-05-15T12:42:09.575 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-05-15T12:42:09.575 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:09.575 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:09.575 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:09.575 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:09.575 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2024-05-15T12:42:09.575 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:09.575 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:09.575 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:09.575 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-05-15T12:42:09.575 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.582+0000 7fb4727fc640 10 monclient: _finish_auth 0 2024-05-15T12:42:09.575 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.582+0000 7fb4727fc640 10 monclient: _check_auth_tickets 2024-05-15T12:42:09.575 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.582+0000 7fb4727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:42:09.576 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.582+0000 7fb47953a640 5 monclient: authenticate success, global_id 4162 2024-05-15T12:42:09.576 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.582+0000 7fb47953a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:42:09.576 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.584+0000 7fb4727fc640 1 -- 172.21.15.178:0/768645901 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb464037070 con 0x7fb474070aa0 2024-05-15T12:42:09.576 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.584+0000 7fb4727fc640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:42:09.576 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.584+0000 7fb4727fc640 1 -- 172.21.15.178:0/768645901 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb464031940 con 0x7fb474070aa0 2024-05-15T12:42:09.576 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.584+0000 7fb4727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:09.576 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.584+0000 7fb4727fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:42:09.576 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.584+0000 7fb4727fc640 10 monclient: dump: 2024-05-15T12:42:09.576 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-05-15T12:42:09.576 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:09.576 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:09.576 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:09.576 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:09.576 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2024-05-15T12:42:09.576 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:09.576 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:09.577 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:09.577 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-05-15T12:42:09.577 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.584+0000 7fb47953a640 10 monclient: get_monmap_and_config success 2024-05-15T12:42:09.577 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.584+0000 7fb47953a640 4 set_mon_vals no callback set 2024-05-15T12:42:09.581 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.585+0000 7fb47953a640 10 monclient: shutdown 2024-05-15T12:42:09.581 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.585+0000 7fb47953a640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:42:09.581 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.585+0000 7fb47953a640 1 -- 172.21.15.178:0/768645901 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb474070aa0 msgr2=0x7fb474140810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:09.582 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.585+0000 7fb47953a640 1 --2- 172.21.15.178:0/768645901 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb474070aa0 0x7fb474140810 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb4640097a0 tx=0x7fb46402b840 comp rx=0 tx=0).stop 2024-05-15T12:42:09.582 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.585+0000 7fb47953a640 1 -- 172.21.15.178:0/768645901 shutdown_connections 2024-05-15T12:42:09.582 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.585+0000 7fb47953a640 1 --2- 172.21.15.178:0/768645901 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb474140d50 0x7fb4741439d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:09.582 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.585+0000 7fb47953a640 1 --2- 172.21.15.178:0/768645901 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb474070aa0 0x7fb474140810 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:09.582 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.585+0000 7fb47953a640 1 -- 172.21.15.178:0/768645901 >> 172.21.15.178:0/768645901 conn(0x7fb4740b5f40 msgr2=0x7fb4740b6350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:42:09.582 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.585+0000 7fb47953a640 1 -- 172.21.15.178:0/768645901 shutdown_connections 2024-05-15T12:42:09.582 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.585+0000 7fb47953a640 1 -- 172.21.15.178:0/768645901 wait complete. 2024-05-15T12:42:09.582 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.586+0000 7fb47953a640 10 monclient: build_initial_monmap 2024-05-15T12:42:09.582 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.586+0000 7fb47953a640 10 monclient: monmap: 2024-05-15T12:42:09.582 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 0 2024-05-15T12:42:09.582 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:42:09.582 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 0.000000 2024-05-15T12:42:09.582 INFO:teuthology.orchestra.run.smithi178.stderr:created 0.000000 2024-05-15T12:42:09.582 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 0 (unknown) 2024-05-15T12:42:09.582 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2024-05-15T12:42:09.583 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T12:42:09.583 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T12:42:09.583 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T12:42:09.583 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-05-15T12:42:09.583 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.586+0000 7fb47953a640 1 librados: starting msgr at 2024-05-15T12:42:09.583 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.586+0000 7fb47953a640 1 librados: starting objecter 2024-05-15T12:42:09.583 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.586+0000 7fb47953a640 1 Processor -- start 2024-05-15T12:42:09.583 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.586+0000 7fb47953a640 1 -- start start 2024-05-15T12:42:09.583 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.586+0000 7fb47953a640 1 librados: setting wanted keys 2024-05-15T12:42:09.583 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.586+0000 7fb47953a640 1 librados: calling monclient init 2024-05-15T12:42:09.583 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.586+0000 7fb47953a640 10 monclient: init 2024-05-15T12:42:09.583 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.587+0000 7fb47953a640 10 monclient: _reopen_session rank -1 2024-05-15T12:42:09.583 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.587+0000 7fb47953a640 10 monclient: _add_conns ranks=[2,0,1] 2024-05-15T12:42:09.583 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.587+0000 7fb47953a640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb474070aa0 0x7fb4741d8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:09.583 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.587+0000 7fb47953a640 10 monclient(hunting): picked mon.noname-c con 0x7fb474070aa0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:42:09.583 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.587+0000 7fb47953a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb474140d50 0x7fb474139e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:09.584 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.587+0000 7fb47953a640 10 monclient(hunting): picked mon.noname-a con 0x7fb474140d50 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T12:42:09.584 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.587+0000 7fb47953a640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb474143f10 0x7fb47413a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:09.584 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.587+0000 7fb47953a640 10 monclient(hunting): picked mon.noname-b con 0x7fb474143f10 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:42:09.584 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.587+0000 7fb47953a640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:09.584 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.587+0000 7fb47953a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb474141340 con 0x7fb474140d50 2024-05-15T12:42:09.584 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.587+0000 7fb47953a640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:09.584 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.587+0000 7fb47953a640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4741411c0 con 0x7fb474143f10 2024-05-15T12:42:09.584 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.587+0000 7fb47953a640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:09.584 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.587+0000 7fb47953a640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb474070e80 con 0x7fb474070aa0 2024-05-15T12:42:09.584 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.587+0000 7fb47953a640 10 monclient(hunting): _renew_subs 2024-05-15T12:42:09.584 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.587+0000 7fb47953a640 10 monclient(hunting): authenticate will time out at 1285.924651s 2024-05-15T12:42:09.584 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.587+0000 7fb46bfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb474140d50 0x7fb474139e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:09.584 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.587+0000 7fb4737fe640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb474143f10 0x7fb47413a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:09.584 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.587+0000 7fb472ffd640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb474070aa0 0x7fb4741d8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:09.585 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.588+0000 7fb4737fe640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb474143f10 0x7fb47413a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.178:45010/0 (socket says 172.21.15.178:45010) 2024-05-15T12:42:09.585 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.588+0000 7fb4737fe640 1 -- 172.21.15.178:0/2411502571 learned_addr learned my addr 172.21.15.178:0/2411502571 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-05-15T12:42:09.585 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.588+0000 7fb4737fe640 10 monclient(hunting): get_auth_request con 0x7fb474143f10 auth_method 0 2024-05-15T12:42:09.585 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.588+0000 7fb4737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:09.585 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.588+0000 7fb4737fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:09.585 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.588+0000 7fb4737fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:09.585 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.588+0000 7fb46bfff640 10 monclient(hunting): get_auth_request con 0x7fb474140d50 auth_method 0 2024-05-15T12:42:09.585 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.588+0000 7fb46bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:09.585 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.588+0000 7fb46bfff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:09.585 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.588+0000 7fb46bfff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:09.585 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.588+0000 7fb472ffd640 10 monclient(hunting): get_auth_request con 0x7fb474070aa0 auth_method 0 2024-05-15T12:42:09.585 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.588+0000 7fb472ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:09.585 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.588+0000 7fb472ffd640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:09.585 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.588+0000 7fb472ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:09.585 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.588+0000 7fb4737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:09.585 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.588+0000 7fb4737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:09.585 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.588+0000 7fb4737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:09.586 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.588+0000 7fb46bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:09.586 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.588+0000 7fb46bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:09.586 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.588+0000 7fb46bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:09.586 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.588+0000 7fb472ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:09.586 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.588+0000 7fb472ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:09.586 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.588+0000 7fb472ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:09.586 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.591+0000 7fb46bfff640 10 monclient(hunting): handle_auth_done global_id 4158 payload 824 2024-05-15T12:42:09.586 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.591+0000 7fb46bfff640 1 -- 172.21.15.178:0/2411502571 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb474070aa0 msgr2=0x7fb4741d8cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:09.586 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.591+0000 7fb46bfff640 1 --2- 172.21.15.178:0/2411502571 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb474070aa0 0x7fb4741d8cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:09.586 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.591+0000 7fb46bfff640 1 -- 172.21.15.178:0/2411502571 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb474143f10 msgr2=0x7fb47413a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:09.586 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.591+0000 7fb46bfff640 1 --2- 172.21.15.178:0/2411502571 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb474143f10 0x7fb47413a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:09.586 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.591+0000 7fb46bfff640 10 monclient: _finish_hunting 0 2024-05-15T12:42:09.586 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.591+0000 7fb46bfff640 1 monclient: found mon.noname-a 2024-05-15T12:42:09.586 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.591+0000 7fb46bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:42:09.586 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.591+0000 7fb46bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.117:3300/0 2024-05-15T12:42:09.587 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.591+0000 7fb46bfff640 1 -- 172.21.15.178:0/2411502571 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4740af6d0 con 0x7fb474140d50 2024-05-15T12:42:09.587 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.591+0000 7fb4737fe640 1 --2- 172.21.15.178:0/2411502571 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb474143f10 0x7fb47413a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-15T12:42:09.587 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-05-15T12:42:09.592+0000 7f5134c38040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-15T12:42:09.610 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.621+0000 7fb46bfff640 1 --2- 172.21.15.178:0/2411502571 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb474140d50 0x7fb474139e20 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb45c00e850 tx=0x7fb45c00ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:42:09.611 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.621+0000 7fb46b7fe640 10 client.?.objecter ms_handle_connect 0x7fb474140d50 2024-05-15T12:42:09.611 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.621+0000 7fb46b7fe640 10 client.?.objecter resend_mon_ops 2024-05-15T12:42:09.644 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-05-15T12:42:09.654+0000 7f18296bb040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-15T12:42:09.661 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.670+0000 7fb46b7fe640 1 -- 172.21.15.178:0/2411502571 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb45c017070 con 0x7fb474140d50 2024-05-15T12:42:09.661 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.670+0000 7fb46b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:09.662 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.670+0000 7fb46b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T12:42:09.662 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.670+0000 7fb46b7fe640 10 monclient: dump: 2024-05-15T12:42:09.662 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-05-15T12:42:09.662 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:09.662 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:09.662 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:09.662 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:09.662 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2024-05-15T12:42:09.662 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:09.662 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:09.662 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:09.662 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-05-15T12:42:09.663 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.670+0000 7fb46b7fe640 10 monclient: _finish_auth 0 2024-05-15T12:42:09.663 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.670+0000 7fb46b7fe640 10 monclient: _check_auth_tickets 2024-05-15T12:42:09.663 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.670+0000 7fb46b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:42:09.663 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.670+0000 7fb46b7fe640 1 -- 172.21.15.178:0/2411502571 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb45c00c940 con 0x7fb474140d50 2024-05-15T12:42:09.663 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.670+0000 7fb47953a640 5 monclient: authenticate success, global_id 4158 2024-05-15T12:42:09.663 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.670+0000 7fb47953a640 10 monclient: _renew_subs 2024-05-15T12:42:09.663 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.670+0000 7fb47953a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:42:09.663 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.670+0000 7fb47953a640 1 -- 172.21.15.178:0/2411502571 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4740ad070 con 0x7fb474140d50 2024-05-15T12:42:09.663 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.670+0000 7fb46b7fe640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:42:09.663 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.670+0000 7fb46b7fe640 1 -- 172.21.15.178:0/2411502571 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb45c00cd10 con 0x7fb474140d50 2024-05-15T12:42:09.663 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.670+0000 7fb46b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:09.663 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.670+0000 7fb46b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T12:42:09.663 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.670+0000 7fb46b7fe640 10 monclient: dump: 2024-05-15T12:42:09.663 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-05-15T12:42:09.663 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:09.664 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:09.664 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:09.664 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:09.664 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2024-05-15T12:42:09.664 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:09.664 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:09.664 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:09.664 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-05-15T12:42:09.664 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.670+0000 7fb471ffb640 4 set_mon_vals no callback set 2024-05-15T12:42:09.664 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.670+0000 7fb47953a640 10 client.4158.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:42:09.664 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.670+0000 7fb47953a640 10 monclient: _renew_subs 2024-05-15T12:42:09.664 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.670+0000 7fb47953a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:42:09.664 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.670+0000 7fb47953a640 1 -- 172.21.15.178:0/2411502571 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4741d96e0 con 0x7fb474140d50 2024-05-15T12:42:09.664 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.670+0000 7fb47953a640 1 librados: init done 2024-05-15T12:42:09.664 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.671+0000 7fb47953a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:42:09.664 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.671+0000 7fb47953a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:42:09.665 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.671+0000 7fb47953a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:42:09.665 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.671+0000 7fb47953a640 1 -- 172.21.15.178:0/2411502571 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb47404f0e0 con 0x7fb474140d50 2024-05-15T12:42:09.757 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.767+0000 7fb46b7fe640 1 -- 172.21.15.178:0/2411502571 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fb45c01a070 con 0x7fb474140d50 2024-05-15T12:42:09.761 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.771+0000 7fb46b7fe640 1 -- 172.21.15.178:0/2411502571 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7fb45c006560 con 0x7fb474140d50 2024-05-15T12:42:09.761 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.771+0000 7fb46b7fe640 10 client.4158.objecter ms_dispatch 0x7fb474000b90 osd_map(9..9 src has 1..9) v4 2024-05-15T12:42:09.761 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.771+0000 7fb46b7fe640 3 client.4158.objecter handle_osd_map got epochs [9,9] > 0 2024-05-15T12:42:09.761 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.771+0000 7fb46b7fe640 3 client.4158.objecter handle_osd_map decoding full epoch 9 2024-05-15T12:42:09.762 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.771+0000 7fb46b7fe640 20 client.4158.objecter dump_active .. 0 homeless 2024-05-15T12:42:09.762 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.771+0000 7fb46b7fe640 1 -- 172.21.15.178:0/2411502571 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fb45c006760 con 0x7fb474140d50 2024-05-15T12:42:09.762 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-05-15T12:42:09.771+0000 7f18296bb040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-15T12:42:09.762 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.771+0000 7fb46b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:42:09.762 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.771+0000 7fb46b7fe640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:42:09.843 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.853+0000 7fb47953a640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "323233c7-a3af-4fb0-ac70-de523a279b44", "id": 7}] 2024-05-15T12:42:09.843 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.853+0000 7fb47953a640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "323233c7-a3af-4fb0-ac70-de523a279b44", "id": 7}] 2024-05-15T12:42:09.843 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.853+0000 7fb47953a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:42:09.843 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.853+0000 7fb47953a640 1 -- 172.21.15.178:0/2411502571 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd new", "uuid": "323233c7-a3af-4fb0-ac70-de523a279b44", "id": 7} v 0) v1 -- 0x7fb4740afcd0 con 0x7fb474140d50 2024-05-15T12:42:09.862 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:09.872+0000 7fb46b7fe640 1 -- 172.21.15.178:0/2411502571 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mgrmap(e 2) v1 ==== 44914+0+0 (secure 0 0 0) 0x7fb45c005990 con 0x7fb474140d50 2024-05-15T12:42:10.282 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:10.292+0000 7fb46b7fe640 1 -- 172.21.15.178:0/2411502571 <== mon.0 v2:172.21.15.117:3300/0 8 ==== mon_command_ack([{"prefix": "osd new", "uuid": "323233c7-a3af-4fb0-ac70-de523a279b44", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb45c00fcc0 con 0x7fb474140d50 2024-05-15T12:42:10.283 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:10.292+0000 7fb46b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "323233c7-a3af-4fb0-ac70-de523a279b44", "id": 7}] 2024-05-15T12:42:10.283 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:10.292+0000 7fb46b7fe640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:42:10.283 INFO:teuthology.orchestra.run.smithi178.stdout:7 2024-05-15T12:42:10.284 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:10.293+0000 7fb47953a640 10 librados: watch_flush enter 2024-05-15T12:42:10.284 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:10.293+0000 7fb47953a640 10 librados: watch_flush exit 2024-05-15T12:42:10.284 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:10.293+0000 7fb47953a640 20 client.4158.objecter shutdown clearing up homeless session... 2024-05-15T12:42:10.284 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:10.293+0000 7fb47953a640 10 client.4158.objecter successfully canceled tick 2024-05-15T12:42:10.284 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:10.293+0000 7fb47953a640 10 monclient: shutdown 2024-05-15T12:42:10.285 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:10.293+0000 7fb47953a640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:42:10.285 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:10.293+0000 7fb47953a640 1 -- 172.21.15.178:0/2411502571 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb474140d50 msgr2=0x7fb474139e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:10.285 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:10.293+0000 7fb47953a640 1 --2- 172.21.15.178:0/2411502571 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb474140d50 0x7fb474139e20 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb45c00e850 tx=0x7fb45c00ed10 comp rx=0 tx=0).stop 2024-05-15T12:42:10.285 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:10.293+0000 7fb47953a640 1 -- 172.21.15.178:0/2411502571 shutdown_connections 2024-05-15T12:42:10.285 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:10.293+0000 7fb47953a640 1 --2- 172.21.15.178:0/2411502571 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb474143f10 0x7fb47413a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:10.285 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:10.293+0000 7fb47953a640 1 --2- 172.21.15.178:0/2411502571 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb474140d50 0x7fb474139e20 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:10.285 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:10.293+0000 7fb47953a640 1 --2- 172.21.15.178:0/2411502571 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb474070aa0 0x7fb4741d8cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:10.285 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:10.293+0000 7fb47953a640 1 -- 172.21.15.178:0/2411502571 >> 172.21.15.178:0/2411502571 conn(0x7fb4740b5f40 msgr2=0x7fb4740b5510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:42:10.285 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:10.293+0000 7fb47953a640 1 -- 172.21.15.178:0/2411502571 shutdown_connections 2024-05-15T12:42:10.285 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:10.294+0000 7fb47953a640 1 -- 172.21.15.178:0/2411502571 wait complete. 2024-05-15T12:42:10.285 INFO:teuthology.orchestra.run.smithi178.stderr:2024-05-15T12:42:10.294+0000 7fb47953a640 1 librados: shutdown 2024-05-15T12:42:10.294 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-05-15T12:42:10.295 INFO:tasks.ceph.osd.0:Restarting daemon 2024-05-15T12:42:10.295 DEBUG:teuthology.orchestra.run.smithi117:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-05-15T12:42:10.297 INFO:tasks.ceph.osd.0:Started 2024-05-15T12:42:10.297 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-05-15T12:42:10.297 INFO:tasks.ceph.osd.1:Restarting daemon 2024-05-15T12:42:10.297 DEBUG:teuthology.orchestra.run.smithi117:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-05-15T12:42:10.302 INFO:tasks.ceph.osd.1:Started 2024-05-15T12:42:10.302 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-05-15T12:42:10.302 INFO:tasks.ceph.osd.2:Restarting daemon 2024-05-15T12:42:10.302 DEBUG:teuthology.orchestra.run.smithi117:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-05-15T12:42:10.304 INFO:tasks.ceph.osd.2:Started 2024-05-15T12:42:10.304 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-05-15T12:42:10.304 INFO:tasks.ceph.osd.3:Restarting daemon 2024-05-15T12:42:10.305 DEBUG:teuthology.orchestra.run.smithi117:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-05-15T12:42:10.307 INFO:tasks.ceph.osd.3:Started 2024-05-15T12:42:10.307 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-05-15T12:42:10.307 INFO:tasks.ceph.osd.4:Restarting daemon 2024-05-15T12:42:10.307 DEBUG:teuthology.orchestra.run.smithi178:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-05-15T12:42:10.309 INFO:tasks.ceph.osd.4:Started 2024-05-15T12:42:10.309 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-05-15T12:42:10.310 INFO:tasks.ceph.osd.5:Restarting daemon 2024-05-15T12:42:10.310 DEBUG:teuthology.orchestra.run.smithi178:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-05-15T12:42:10.312 INFO:tasks.ceph.osd.5:Started 2024-05-15T12:42:10.312 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-05-15T12:42:10.312 INFO:tasks.ceph.osd.6:Restarting daemon 2024-05-15T12:42:10.312 DEBUG:teuthology.orchestra.run.smithi178:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-05-15T12:42:10.314 INFO:tasks.ceph.osd.6:Started 2024-05-15T12:42:10.314 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-05-15T12:42:10.314 INFO:tasks.ceph.osd.7:Restarting daemon 2024-05-15T12:42:10.314 DEBUG:teuthology.orchestra.run.smithi178:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-05-15T12:42:10.317 INFO:tasks.ceph.osd.7:Started 2024-05-15T12:42:10.317 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-15T12:42:10.448 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.457+0000 7f6afae85640 10 monclient: get_monmap_and_config 2024-05-15T12:42:10.448 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.457+0000 7f6afae85640 10 monclient: build_initial_monmap 2024-05-15T12:42:10.448 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.457+0000 7f6afae85640 10 monclient: monmap: 2024-05-15T12:42:10.448 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:42:10.448 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:10.448 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:42:10.458726+0000 2024-05-15T12:42:10.448 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:42:10.458726+0000 2024-05-15T12:42:10.448 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:42:10.448 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:42:10.448 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T12:42:10.448 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T12:42:10.448 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T12:42:10.449 INFO:teuthology.orchestra.run.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T12:42:10.449 INFO:teuthology.orchestra.run.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T12:42:10.449 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:42:10.450 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.458+0000 7f6afae85640 1 Processor -- start 2024-05-15T12:42:10.450 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.458+0000 7f6afae85640 1 -- start start 2024-05-15T12:42:10.450 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.458+0000 7f6afae85640 10 monclient: init 2024-05-15T12:42:10.450 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.459+0000 7f6afae85640 10 monclient: _reopen_session rank -1 2024-05-15T12:42:10.450 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.459+0000 7f6afae85640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-05-15T12:42:10.450 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.459+0000 7f6afae85640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6af40b86e0 0x7f6af40aa0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:10.451 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.459+0000 7f6afae85640 10 monclient(hunting): picked mon.noname-b con 0x7f6af40b86e0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:42:10.451 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.459+0000 7f6afae85640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6af40b1dd0 addr v1:172.21.15.178:6789/0 2024-05-15T12:42:10.451 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.459+0000 7f6afae85640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6af413f870 addr v1:172.21.15.117:6789/0 2024-05-15T12:42:10.451 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.459+0000 7f6afae85640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6af40a9bc0 con 0x7f6af413f870 2024-05-15T12:42:10.451 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.459+0000 7f6afae85640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6af40a9850 con 0x7f6af40b1dd0 2024-05-15T12:42:10.451 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.459+0000 7f6afae85640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:10.451 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.459+0000 7f6afae85640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6af40b8ac0 con 0x7f6af40b86e0 2024-05-15T12:42:10.451 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.459+0000 7f6afae85640 10 monclient(hunting): _renew_subs 2024-05-15T12:42:10.451 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.459+0000 7f6afae85640 10 monclient(hunting): authenticate will time out at 1293.701191s 2024-05-15T12:42:10.460 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.470+0000 7f6af8c21640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6af40b86e0 0x7f6af40aa0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:10.461 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.471+0000 7f6af8c21640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6af40b86e0 0x7f6af40aa0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.117:46662/0 (socket says 172.21.15.117:46662) 2024-05-15T12:42:10.461 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.471+0000 7f6af8c21640 1 -- 172.21.15.117:0/3531695196 learned_addr learned my addr 172.21.15.117:0/3531695196 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:42:10.461 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.471+0000 7f6af8c21640 10 monclient(hunting): get_auth_request con 0x7f6af40b86e0 auth_method 0 2024-05-15T12:42:10.461 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.471+0000 7f6af8c21640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:10.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.471+0000 7f6af8c21640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:10.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.471+0000 7f6af8c21640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:10.480 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.490+0000 7f6af8c21640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:10.480 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.490+0000 7f6af8c21640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:10.480 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.490+0000 7f6af8c21640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:10.483 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.493+0000 7f6af8c21640 10 monclient(hunting): handle_auth_done global_id 4154 payload 274 2024-05-15T12:42:10.483 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.493+0000 7f6af8c21640 1 -- 172.21.15.117:0/3531695196 >> v1:172.21.15.178:6789/0 conn(0x7f6af40b1dd0 legacy=0x7f6af4145bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:10.483 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.493+0000 7f6af8c21640 1 -- 172.21.15.117:0/3531695196 >> v1:172.21.15.117:6789/0 conn(0x7f6af413f870 legacy=0x7f6af413fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:42:10.483 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.493+0000 7f6af8c21640 10 monclient: _finish_hunting 0 2024-05-15T12:42:10.484 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.493+0000 7f6af8c21640 1 monclient: found mon.noname-b 2024-05-15T12:42:10.484 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.493+0000 7f6af8c21640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:42:10.484 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.493+0000 7f6af8c21640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.117:3301/0 2024-05-15T12:42:10.484 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.493+0000 7f6af8c21640 1 -- 172.21.15.117:0/3531695196 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6af40b3a60 con 0x7f6af40b86e0 2024-05-15T12:42:10.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.495+0000 7f6af8c21640 1 --2- 172.21.15.117:0/3531695196 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6af40b86e0 0x7f6af40aa0e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6af406cae0 tx=0x7f6ae802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47c4d04e757db268 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:42:10.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.501+0000 7f6af37fe640 1 -- 172.21.15.117:0/3531695196 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ae8035020 con 0x7f6af40b86e0 2024-05-15T12:42:10.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.501+0000 7f6af37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:10.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.501+0000 7f6af37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:42:10.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.501+0000 7f6af37fe640 10 monclient: dump: 2024-05-15T12:42:10.492 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:42:10.492 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:10.492 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:10.492 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:10.492 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:10.492 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:42:10.492 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:10.492 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:10.492 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:10.492 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:42:10.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.501+0000 7f6af37fe640 10 monclient: _finish_auth 0 2024-05-15T12:42:10.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.501+0000 7f6af37fe640 10 monclient: _check_auth_tickets 2024-05-15T12:42:10.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.501+0000 7f6af37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:42:10.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.501+0000 7f6af37fe640 1 -- 172.21.15.117:0/3531695196 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ae8031830 con 0x7f6af40b86e0 2024-05-15T12:42:10.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.501+0000 7f6af37fe640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:42:10.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.501+0000 7f6af37fe640 1 -- 172.21.15.117:0/3531695196 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ae8031c00 con 0x7f6af40b86e0 2024-05-15T12:42:10.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.501+0000 7f6af37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:10.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.501+0000 7f6af37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T12:42:10.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.501+0000 7f6af37fe640 10 monclient: dump: 2024-05-15T12:42:10.493 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:42:10.493 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:10.493 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:10.493 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:10.493 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:10.493 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:42:10.493 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:10.494 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:10.494 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:10.494 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:42:10.494 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.501+0000 7f6afae85640 5 monclient: authenticate success, global_id 4154 2024-05-15T12:42:10.494 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.501+0000 7f6afae85640 10 monclient: get_monmap_and_config success 2024-05-15T12:42:10.494 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.501+0000 7f6afae85640 4 set_mon_vals no callback set 2024-05-15T12:42:10.496 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.502+0000 7f6afae85640 10 monclient: shutdown 2024-05-15T12:42:10.496 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.502+0000 7f6afae85640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:42:10.497 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.502+0000 7f6afae85640 1 -- 172.21.15.117:0/3531695196 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6af40b86e0 msgr2=0x7f6af40aa0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:10.497 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.502+0000 7f6afae85640 1 --2- 172.21.15.117:0/3531695196 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6af40b86e0 0x7f6af40aa0e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6af406cae0 tx=0x7f6ae802b8d0 comp rx=0 tx=0).stop 2024-05-15T12:42:10.497 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.502+0000 7f6afae85640 1 -- 172.21.15.117:0/3531695196 shutdown_connections 2024-05-15T12:42:10.497 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.502+0000 7f6afae85640 1 --2- 172.21.15.117:0/3531695196 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6af40b86e0 0x7f6af40aa0e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:10.497 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.502+0000 7f6afae85640 1 -- 172.21.15.117:0/3531695196 >> 172.21.15.117:0/3531695196 conn(0x7f6af4066f00 msgr2=0x7f6af4067310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:42:10.497 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.502+0000 7f6afae85640 1 -- 172.21.15.117:0/3531695196 shutdown_connections 2024-05-15T12:42:10.497 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.502+0000 7f6afae85640 1 -- 172.21.15.117:0/3531695196 wait complete. 2024-05-15T12:42:10.497 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.503+0000 7f6afae85640 10 monclient: build_initial_monmap 2024-05-15T12:42:10.497 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.503+0000 7f6afae85640 10 monclient: monmap: 2024-05-15T12:42:10.497 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:42:10.497 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:42:10.497 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-05-15T12:42:10.497 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-05-15T12:42:10.497 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:42:10.498 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:42:10.498 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T12:42:10.498 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T12:42:10.498 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T12:42:10.498 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:42:10.498 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.504+0000 7f6afae85640 1 librados: starting msgr at 2024-05-15T12:42:10.498 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.504+0000 7f6afae85640 1 librados: starting objecter 2024-05-15T12:42:10.498 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.504+0000 7f6afae85640 1 Processor -- start 2024-05-15T12:42:10.498 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.504+0000 7f6afae85640 1 -- start start 2024-05-15T12:42:10.498 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.504+0000 7f6afae85640 1 librados: setting wanted keys 2024-05-15T12:42:10.498 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.504+0000 7f6afae85640 1 librados: calling monclient init 2024-05-15T12:42:10.498 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.504+0000 7f6afae85640 10 monclient: init 2024-05-15T12:42:10.498 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.505+0000 7f6afae85640 10 monclient: _reopen_session rank -1 2024-05-15T12:42:10.498 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.505+0000 7f6afae85640 10 monclient: _add_conns ranks=[1,0,2] 2024-05-15T12:42:10.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.505+0000 7f6afae85640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6af40b1dd0 0x7f6af4138390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:10.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.505+0000 7f6afae85640 10 monclient(hunting): picked mon.noname-b con 0x7f6af40b1dd0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:42:10.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.505+0000 7f6afae85640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6af413f870 0x7f6af41367c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:10.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.505+0000 7f6afae85640 10 monclient(hunting): picked mon.noname-a con 0x7f6af413f870 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T12:42:10.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.505+0000 7f6afae85640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6af4138f70 0x7f6af40b7730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:10.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.506+0000 7f6afae85640 10 monclient(hunting): picked mon.noname-c con 0x7f6af4138f70 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:42:10.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.506+0000 7f6afae85640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:10.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.506+0000 7f6afae85640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6af414e130 con 0x7f6af413f870 2024-05-15T12:42:10.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.506+0000 7f6afae85640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:10.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.506+0000 7f6afae85640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6af40aa620 con 0x7f6af40b1dd0 2024-05-15T12:42:10.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.506+0000 7f6afae85640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:10.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.506+0000 7f6afae85640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6af414e2b0 con 0x7f6af4138f70 2024-05-15T12:42:10.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.506+0000 7f6afae85640 10 monclient(hunting): _renew_subs 2024-05-15T12:42:10.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.506+0000 7f6afae85640 10 monclient(hunting): authenticate will time out at 1293.748162s 2024-05-15T12:42:10.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.506+0000 7f6af9422640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6af4138f70 0x7f6af40b7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:10.501 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.507+0000 7f6af9422640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6af4138f70 0x7f6af40b7730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.117:46670/0 (socket says 172.21.15.117:46670) 2024-05-15T12:42:10.501 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.507+0000 7f6af9422640 1 -- 172.21.15.117:0/4106370562 learned_addr learned my addr 172.21.15.117:0/4106370562 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:42:10.502 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.507+0000 7f6af9422640 10 monclient(hunting): get_auth_request con 0x7f6af4138f70 auth_method 0 2024-05-15T12:42:10.502 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.507+0000 7f6af9422640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:10.502 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.507+0000 7f6af9422640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:10.502 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.507+0000 7f6af9422640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:10.502 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.508+0000 7f6af9422640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:10.502 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.508+0000 7f6af9422640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:10.502 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.508+0000 7f6af9422640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:10.502 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.509+0000 7f6af8c21640 1 --2- 172.21.15.117:0/4106370562 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6af40b1dd0 0x7f6af4138390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:10.502 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.511+0000 7f6af8c21640 10 monclient(hunting): get_auth_request con 0x7f6af40b1dd0 auth_method 0 2024-05-15T12:42:10.502 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.511+0000 7f6af8c21640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:10.502 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.511+0000 7f6af8c21640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:10.502 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.511+0000 7f6af8c21640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:10.502 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.511+0000 7f6af8c21640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:10.503 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.511+0000 7f6af8c21640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:10.503 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.511+0000 7f6af8c21640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:10.504 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.513+0000 7f6af9422640 10 monclient(hunting): handle_auth_done global_id 4157 payload 824 2024-05-15T12:42:10.504 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.513+0000 7f6af9422640 1 -- 172.21.15.117:0/4106370562 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6af40b1dd0 msgr2=0x7f6af4138390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:10.504 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.513+0000 7f6af9422640 1 --2- 172.21.15.117:0/4106370562 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6af40b1dd0 0x7f6af4138390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:10.504 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.513+0000 7f6af9422640 1 -- 172.21.15.117:0/4106370562 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6af413f870 msgr2=0x7f6af41367c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:10.504 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.513+0000 7f6af9422640 1 --2- 172.21.15.117:0/4106370562 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6af413f870 0x7f6af41367c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:10.504 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.513+0000 7f6af9422640 10 monclient: _finish_hunting 0 2024-05-15T12:42:10.504 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.513+0000 7f6af9422640 1 monclient: found mon.noname-c 2024-05-15T12:42:10.504 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.513+0000 7f6af9422640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:42:10.504 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.513+0000 7f6af9422640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.117:3301/0 2024-05-15T12:42:10.504 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.513+0000 7f6af9422640 1 -- 172.21.15.117:0/4106370562 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6af41378d0 con 0x7f6af4138f70 2024-05-15T12:42:10.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.515+0000 7f6af9422640 1 --2- 172.21.15.117:0/4106370562 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6af4138f70 0x7f6af40b7730 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6aec00b990 tx=0x7f6aec00be50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:42:10.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.515+0000 7f6af17fa640 10 client.?.objecter ms_handle_connect 0x7f6af4138f70 2024-05-15T12:42:10.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.515+0000 7f6af17fa640 10 client.?.objecter resend_mon_ops 2024-05-15T12:42:10.510 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.519+0000 7f6af17fa640 1 -- 172.21.15.117:0/4106370562 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aec019070 con 0x7f6af4138f70 2024-05-15T12:42:10.510 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.519+0000 7f6af17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:10.510 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.519+0000 7f6af17fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:42:10.510 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.519+0000 7f6af17fa640 10 monclient: dump: 2024-05-15T12:42:10.510 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:42:10.511 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:10.511 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:10.511 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:10.511 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:10.511 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:42:10.511 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:10.511 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:10.511 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:10.511 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:42:10.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.519+0000 7f6af17fa640 10 monclient: _finish_auth 0 2024-05-15T12:42:10.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.519+0000 7f6af17fa640 10 monclient: _check_auth_tickets 2024-05-15T12:42:10.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.519+0000 7f6af17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:42:10.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.519+0000 7f6afae85640 5 monclient: authenticate success, global_id 4157 2024-05-15T12:42:10.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.519+0000 7f6afae85640 10 monclient: _renew_subs 2024-05-15T12:42:10.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.519+0000 7f6afae85640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T12:42:10.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.519+0000 7f6afae85640 1 -- 172.21.15.117:0/4106370562 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6af4137bb0 con 0x7f6af4138f70 2024-05-15T12:42:10.512 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.519+0000 7f6afae85640 10 client.4157.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:42:10.512 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.519+0000 7f6afae85640 10 monclient: _renew_subs 2024-05-15T12:42:10.512 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.519+0000 7f6afae85640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T12:42:10.512 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.519+0000 7f6afae85640 1 -- 172.21.15.117:0/4106370562 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6af4136d00 con 0x7f6af4138f70 2024-05-15T12:42:10.512 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.519+0000 7f6afae85640 1 librados: init done 2024-05-15T12:42:10.513 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.521+0000 7f6afae85640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:42:10.513 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.521+0000 7f6afae85640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:42:10.513 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.521+0000 7f6afae85640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T12:42:10.513 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.521+0000 7f6afae85640 1 -- 172.21.15.117:0/4106370562 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6af404f0e0 con 0x7f6af4138f70 2024-05-15T12:42:10.514 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.523+0000 7f6af17fa640 1 -- 172.21.15.117:0/4106370562 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6aec00cba0 con 0x7f6af4138f70 2024-05-15T12:42:10.514 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.523+0000 7f6af17fa640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:42:10.514 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.523+0000 7f6af2ffd640 4 set_mon_vals no callback set 2024-05-15T12:42:10.514 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.523+0000 7f6af17fa640 1 -- 172.21.15.117:0/4106370562 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aec023060 con 0x7f6af4138f70 2024-05-15T12:42:10.514 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.523+0000 7f6af17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:10.514 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.523+0000 7f6af17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T12:42:10.514 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.523+0000 7f6af17fa640 10 monclient: dump: 2024-05-15T12:42:10.514 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:42:10.514 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:10.515 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:10.515 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:10.515 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:10.515 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:42:10.515 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:10.515 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:10.515 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:10.515 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:42:10.533 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.544+0000 7f6af17fa640 1 -- 172.21.15.117:0/4106370562 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 2) v1 ==== 44914+0+0 (secure 0 0 0) 0x7f6aec01c070 con 0x7f6af4138f70 2024-05-15T12:42:10.557 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.567+0000 7f6af17fa640 1 -- 172.21.15.117:0/4106370562 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f6aec007530 con 0x7f6af4138f70 2024-05-15T12:42:10.557 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.567+0000 7f6af17fa640 10 client.4157.objecter ms_dispatch 0x7f6af4000b90 osd_map(10..10 src has 1..10) v4 2024-05-15T12:42:10.557 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.567+0000 7f6af17fa640 3 client.4157.objecter handle_osd_map got epochs [10,10] > 0 2024-05-15T12:42:10.557 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.567+0000 7f6af17fa640 3 client.4157.objecter handle_osd_map decoding full epoch 10 2024-05-15T12:42:10.558 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.567+0000 7f6af17fa640 20 client.4157.objecter dump_active .. 0 homeless 2024-05-15T12:42:10.649 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.658+0000 7f6af17fa640 1 -- 172.21.15.117:0/4106370562 <== mon.2 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f6aec01ecf0 con 0x7f6af4138f70 2024-05-15T12:42:10.649 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.658+0000 7f6af17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:42:10.649 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.658+0000 7f6af17fa640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:42:10.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.778+0000 7f6afae85640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:42:10.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.778+0000 7f6afae85640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:42:10.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.778+0000 7f6afae85640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T12:42:10.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.778+0000 7f6afae85640 1 -- 172.21.15.117:0/4106370562 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6af4067d90 con 0x7f6af4138f70 2024-05-15T12:42:10.852 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.862+0000 7f6af17fa640 1 -- 172.21.15.117:0/4106370562 <== mon.2 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f6aec01e7d0 con 0x7f6af4138f70 2024-05-15T12:42:10.852 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.862+0000 7f6af17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:42:10.853 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.862+0000 7f6af17fa640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:42:10.853 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:42:10.853 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":10,"fsid":"8817f6cf-d933-4825-b972-575ef59f5ba9","created":"2024-05-15T12:42:04.264009+0000","modified":"2024-05-15T12:42:09.879357+0000","last_up_change":"0.000000","last_in_change":"2024-05-15T12:42:09.879357+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"aaffd6d9-1a00-4793-9d1f-65a30501bec4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"efd4e8ed-699c-4ccb-b989-cfadc7aaed35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"708e4e3e-7e79-480f-9dbf-7b972c2f2811","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e7b5d7d7-515c-42d9-9c41-b1ca3977865e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa647c2c-942b-4be3-afb9-6dba6b0a8cf3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5f9bc37-4bd6-4643-8f0d-e53c5f2c8782","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d5af075d-3eaa-4cb5-b6e9-6f01c201f2ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"323233c7-a3af-4fb0-ac70-de523a279b44","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-15T12:42:10.854 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.863+0000 7f6afae85640 10 librados: watch_flush enter 2024-05-15T12:42:10.854 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.863+0000 7f6afae85640 10 librados: watch_flush exit 2024-05-15T12:42:10.854 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.863+0000 7f6afae85640 20 client.4157.objecter shutdown clearing up homeless session... 2024-05-15T12:42:10.854 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.863+0000 7f6afae85640 10 client.4157.objecter successfully canceled tick 2024-05-15T12:42:10.854 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.863+0000 7f6afae85640 10 monclient: shutdown 2024-05-15T12:42:10.854 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.863+0000 7f6afae85640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:42:10.854 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.863+0000 7f6afae85640 1 -- 172.21.15.117:0/4106370562 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6af4138f70 msgr2=0x7f6af40b7730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:10.854 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.863+0000 7f6afae85640 1 --2- 172.21.15.117:0/4106370562 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6af4138f70 0x7f6af40b7730 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6aec00b990 tx=0x7f6aec00be50 comp rx=0 tx=0).stop 2024-05-15T12:42:10.854 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.864+0000 7f6afae85640 1 -- 172.21.15.117:0/4106370562 shutdown_connections 2024-05-15T12:42:10.854 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.864+0000 7f6afae85640 1 --2- 172.21.15.117:0/4106370562 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6af4138f70 0x7f6af40b7730 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:10.854 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.864+0000 7f6afae85640 1 --2- 172.21.15.117:0/4106370562 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6af413f870 0x7f6af41367c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:10.855 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.864+0000 7f6afae85640 1 --2- 172.21.15.117:0/4106370562 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6af40b1dd0 0x7f6af4138390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:10.855 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.864+0000 7f6afae85640 1 -- 172.21.15.117:0/4106370562 >> 172.21.15.117:0/4106370562 conn(0x7f6af4066f00 msgr2=0x7f6af40b4740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:42:10.855 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.864+0000 7f6afae85640 1 -- 172.21.15.117:0/4106370562 shutdown_connections 2024-05-15T12:42:10.855 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.864+0000 7f6afae85640 1 -- 172.21.15.117:0/4106370562 wait complete. 2024-05-15T12:42:10.855 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:42:10.864+0000 7f6afae85640 1 librados: shutdown 2024-05-15T12:42:10.869 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-05-15T12:42:10.870 INFO:tasks.ceph:Waiting for OSDs to come up 2024-05-15T12:42:11.449 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-05-15T12:42:11.460+0000 7f50d5859640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-05-15T12:42:11.674 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-05-15T12:42:11.684+0000 7f50d5859640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-05-15T12:42:11.677 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-15T12:42:11.823 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.833+0000 7f26f467e640 10 monclient: get_monmap_and_config 2024-05-15T12:42:11.824 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.833+0000 7f26f467e640 10 monclient: build_initial_monmap 2024-05-15T12:42:11.824 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.833+0000 7f26f467e640 10 monclient: monmap: 2024-05-15T12:42:11.824 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-05-15T12:42:11.824 INFO:teuthology.misc.health.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:11.824 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-05-15T12:42:11.834692+0000 2024-05-15T12:42:11.824 INFO:teuthology.misc.health.smithi117.stderr:created 2024-05-15T12:42:11.834692+0000 2024-05-15T12:42:11.824 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:42:11.824 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-05-15T12:42:11.824 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T12:42:11.824 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T12:42:11.824 INFO:teuthology.misc.health.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T12:42:11.824 INFO:teuthology.misc.health.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T12:42:11.825 INFO:teuthology.misc.health.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T12:42:11.825 INFO:teuthology.misc.health.smithi117.stderr: 2024-05-15T12:42:11.829 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.834+0000 7f26f467e640 1 Processor -- start 2024-05-15T12:42:11.829 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.834+0000 7f26f467e640 1 -- start start 2024-05-15T12:42:11.829 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.834+0000 7f26f467e640 10 monclient: init 2024-05-15T12:42:11.829 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.835+0000 7f26f467e640 10 monclient: _reopen_session rank -1 2024-05-15T12:42:11.829 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.835+0000 7f26f467e640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-05-15T12:42:11.829 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.835+0000 7f26f467e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f26ec0b54b0 addr v1:172.21.15.178:6789/0 2024-05-15T12:42:11.829 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.835+0000 7f26f467e640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f26ec142db0 0x7f26ec0a9980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:11.829 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.835+0000 7f26f467e640 10 monclient(hunting): picked mon.noname-c con 0x7f26ec142db0 addr v2:172.21.15.178:3300/0 2024-05-15T12:42:11.829 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.835+0000 7f26f467e640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f26ec13f850 0x7f26ec13fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:11.830 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.835+0000 7f26f467e640 10 monclient(hunting): picked mon.noname-a con 0x7f26ec13f850 addr v2:172.21.15.117:3300/0 2024-05-15T12:42:11.830 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.835+0000 7f26f467e640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26ec0ab770 con 0x7f26ec0b54b0 2024-05-15T12:42:11.830 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.835+0000 7f26f467e640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:11.830 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.835+0000 7f26f467e640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26ec0b5890 con 0x7f26ec13f850 2024-05-15T12:42:11.830 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.835+0000 7f26f467e640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:11.830 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.835+0000 7f26f467e640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26ec0aa050 con 0x7f26ec142db0 2024-05-15T12:42:11.830 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.835+0000 7f26f467e640 10 monclient(hunting): _renew_subs 2024-05-15T12:42:11.830 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.835+0000 7f26f467e640 10 monclient(hunting): authenticate will time out at 1295.077739s 2024-05-15T12:42:11.830 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.836+0000 7f26f1c19640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f26ec142db0 0x7f26ec0a9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:11.830 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.837+0000 7f26f1c19640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f26ec142db0 0x7f26ec0a9980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:43236/0 (socket says 172.21.15.117:43236) 2024-05-15T12:42:11.830 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.837+0000 7f26f1c19640 1 -- 172.21.15.117:0/2802031894 learned_addr learned my addr 172.21.15.117:0/2802031894 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:42:11.830 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.837+0000 7f26f1c19640 10 monclient(hunting): get_auth_request con 0x7f26ec142db0 auth_method 0 2024-05-15T12:42:11.830 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.837+0000 7f26f1c19640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:11.830 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.837+0000 7f26f1c19640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:11.830 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.837+0000 7f26f1c19640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:11.831 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.838+0000 7f26f1c19640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:11.831 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.838+0000 7f26f1c19640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:11.831 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.838+0000 7f26f1c19640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:11.832 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.841+0000 7f26f1c19640 10 monclient(hunting): handle_auth_done global_id 4183 payload 274 2024-05-15T12:42:11.832 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.841+0000 7f26f1c19640 1 -- 172.21.15.117:0/2802031894 >> v2:172.21.15.117:3300/0 conn(0x7f26ec13f850 msgr2=0x7f26ec13fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:42:11.832 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.841+0000 7f26f1c19640 1 --2- 172.21.15.117:0/2802031894 >> v2:172.21.15.117:3300/0 conn(0x7f26ec13f850 0x7f26ec13fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:11.832 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.841+0000 7f26f1c19640 1 -- 172.21.15.117:0/2802031894 >> v1:172.21.15.178:6789/0 conn(0x7f26ec0b54b0 legacy=0x7f26ec1426a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:11.832 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.841+0000 7f26f1c19640 10 monclient: _finish_hunting 0 2024-05-15T12:42:11.832 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.841+0000 7f26f1c19640 1 monclient: found mon.noname-c 2024-05-15T12:42:11.832 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.841+0000 7f26f1c19640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:42:11.832 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.841+0000 7f26f1c19640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.178:3300/0 2024-05-15T12:42:11.832 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.841+0000 7f26f1c19640 1 -- 172.21.15.117:0/2802031894 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26ec0aa570 con 0x7f26ec142db0 2024-05-15T12:42:11.837 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.847+0000 7f26f1c19640 1 --2- 172.21.15.117:0/2802031894 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f26ec142db0 0x7f26ec0a9980 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f26ec0b5c60 tx=0x7f26dc02b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=19fd8931407e591f server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:42:11.847 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.853+0000 7f26f1418640 1 -- 172.21.15.117:0/2802031894 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26dc035020 con 0x7f26ec142db0 2024-05-15T12:42:11.847 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.853+0000 7f26f1418640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:11.847 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.853+0000 7f26f1418640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:42:11.847 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.853+0000 7f26f1418640 10 monclient: dump: 2024-05-15T12:42:11.847 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-05-15T12:42:11.847 INFO:teuthology.misc.health.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:11.847 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:11.847 INFO:teuthology.misc.health.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:11.847 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:11.847 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-05-15T12:42:11.848 INFO:teuthology.misc.health.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:11.848 INFO:teuthology.misc.health.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:11.848 INFO:teuthology.misc.health.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:11.848 INFO:teuthology.misc.health.smithi117.stderr: 2024-05-15T12:42:11.848 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.853+0000 7f26f1418640 10 monclient: _finish_auth 0 2024-05-15T12:42:11.848 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.853+0000 7f26f1418640 10 monclient: _check_auth_tickets 2024-05-15T12:42:11.848 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.853+0000 7f26f1418640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:42:11.848 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.853+0000 7f26f1418640 1 -- 172.21.15.117:0/2802031894 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26dc031870 con 0x7f26ec142db0 2024-05-15T12:42:11.848 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.853+0000 7f26f1418640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:42:11.848 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.853+0000 7f26f1418640 1 -- 172.21.15.117:0/2802031894 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26dc031c40 con 0x7f26ec142db0 2024-05-15T12:42:11.848 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.853+0000 7f26f1418640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:11.848 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.853+0000 7f26f1418640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:42:11.848 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.853+0000 7f26f1418640 10 monclient: dump: 2024-05-15T12:42:11.848 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-05-15T12:42:11.848 INFO:teuthology.misc.health.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:11.848 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:11.849 INFO:teuthology.misc.health.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:11.849 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:11.849 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-05-15T12:42:11.849 INFO:teuthology.misc.health.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:11.849 INFO:teuthology.misc.health.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:11.849 INFO:teuthology.misc.health.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:11.849 INFO:teuthology.misc.health.smithi117.stderr: 2024-05-15T12:42:11.849 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.854+0000 7f26f467e640 5 monclient: authenticate success, global_id 4183 2024-05-15T12:42:11.849 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.854+0000 7f26f467e640 10 monclient: get_monmap_and_config success 2024-05-15T12:42:11.849 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.854+0000 7f26f467e640 4 set_mon_vals no callback set 2024-05-15T12:42:11.849 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.854+0000 7f26f467e640 10 monclient: shutdown 2024-05-15T12:42:11.849 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.854+0000 7f26f467e640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:42:11.849 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.854+0000 7f26f467e640 1 -- 172.21.15.117:0/2802031894 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f26ec142db0 msgr2=0x7f26ec0a9980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:11.849 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.854+0000 7f26f467e640 1 --2- 172.21.15.117:0/2802031894 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f26ec142db0 0x7f26ec0a9980 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f26ec0b5c60 tx=0x7f26dc02b910 comp rx=0 tx=0).stop 2024-05-15T12:42:11.849 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.855+0000 7f26f467e640 1 -- 172.21.15.117:0/2802031894 shutdown_connections 2024-05-15T12:42:11.850 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.855+0000 7f26f467e640 1 --2- 172.21.15.117:0/2802031894 >> v2:172.21.15.117:3300/0 conn(0x7f26ec13f850 0x7f26ec13fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:11.850 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.855+0000 7f26f467e640 1 --2- 172.21.15.117:0/2802031894 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f26ec142db0 0x7f26ec0a9980 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:11.850 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.855+0000 7f26f467e640 1 -- 172.21.15.117:0/2802031894 >> 172.21.15.117:0/2802031894 conn(0x7f26ec06d370 msgr2=0x7f26ec06d780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:42:11.850 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.855+0000 7f26f467e640 1 -- 172.21.15.117:0/2802031894 shutdown_connections 2024-05-15T12:42:11.850 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.855+0000 7f26f467e640 1 -- 172.21.15.117:0/2802031894 wait complete. 2024-05-15T12:42:11.850 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.856+0000 7f26f467e640 10 monclient: build_initial_monmap 2024-05-15T12:42:11.850 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.856+0000 7f26f467e640 10 monclient: monmap: 2024-05-15T12:42:11.850 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-05-15T12:42:11.850 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:42:11.850 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-05-15T12:42:11.850 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-05-15T12:42:11.850 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:42:11.850 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-05-15T12:42:11.851 INFO:teuthology.misc.health.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T12:42:11.851 INFO:teuthology.misc.health.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T12:42:11.851 INFO:teuthology.misc.health.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T12:42:11.851 INFO:teuthology.misc.health.smithi117.stderr: 2024-05-15T12:42:11.851 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.856+0000 7f26f467e640 1 librados: starting msgr at 2024-05-15T12:42:11.851 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.856+0000 7f26f467e640 1 librados: starting objecter 2024-05-15T12:42:11.851 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.856+0000 7f26f467e640 1 Processor -- start 2024-05-15T12:42:11.851 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.857+0000 7f26f467e640 1 -- start start 2024-05-15T12:42:11.851 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.857+0000 7f26f467e640 1 librados: setting wanted keys 2024-05-15T12:42:11.851 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.857+0000 7f26f467e640 1 librados: calling monclient init 2024-05-15T12:42:11.851 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.857+0000 7f26f467e640 10 monclient: init 2024-05-15T12:42:11.854 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.858+0000 7f26f467e640 10 monclient: _reopen_session rank -1 2024-05-15T12:42:11.854 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.858+0000 7f26f467e640 10 monclient: _add_conns ranks=[1,0,2] 2024-05-15T12:42:11.854 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.858+0000 7f26f467e640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f26ec0b54b0 0x7f26ec143520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:11.854 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.858+0000 7f26f467e640 10 monclient(hunting): picked mon.noname-b con 0x7f26ec0b54b0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:42:11.854 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.858+0000 7f26f467e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f26ec13f850 0x7f26ec1352d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:11.854 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.858+0000 7f26f467e640 10 monclient(hunting): picked mon.noname-a con 0x7f26ec13f850 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T12:42:11.854 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.858+0000 7f26f467e640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f26ec142db0 0x7f26ec135810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:11.854 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.858+0000 7f26f467e640 10 monclient(hunting): picked mon.noname-c con 0x7f26ec142db0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:42:11.854 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.858+0000 7f26f467e640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:11.854 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.858+0000 7f26f467e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26ec0b4a20 con 0x7f26ec13f850 2024-05-15T12:42:11.855 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.858+0000 7f26f467e640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:11.855 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.858+0000 7f26f467e640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26ec0b48a0 con 0x7f26ec0b54b0 2024-05-15T12:42:11.855 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.858+0000 7f26f467e640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:11.855 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.858+0000 7f26f467e640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26ec14cf70 con 0x7f26ec142db0 2024-05-15T12:42:11.855 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.858+0000 7f26f467e640 10 monclient(hunting): _renew_subs 2024-05-15T12:42:11.855 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.858+0000 7f26f467e640 10 monclient(hunting): authenticate will time out at 1295.100426s 2024-05-15T12:42:11.855 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.858+0000 7f26f241a640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f26ec0b54b0 0x7f26ec143520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:11.855 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.859+0000 7f26f241a640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f26ec0b54b0 0x7f26ec143520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:43244/0 (socket says 172.21.15.117:43244) 2024-05-15T12:42:11.855 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.859+0000 7f26f241a640 1 -- 172.21.15.117:0/3195787842 learned_addr learned my addr 172.21.15.117:0/3195787842 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:42:11.855 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.859+0000 7f26f241a640 10 monclient(hunting): get_auth_request con 0x7f26ec0b54b0 auth_method 0 2024-05-15T12:42:11.855 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.859+0000 7f26f241a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:11.855 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.859+0000 7f26f241a640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:11.855 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.859+0000 7f26f241a640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:11.855 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.860+0000 7f26f2c1b640 1 --2- 172.21.15.117:0/3195787842 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f26ec142db0 0x7f26ec135810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:11.855 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.860+0000 7f26f2c1b640 10 monclient(hunting): get_auth_request con 0x7f26ec142db0 auth_method 0 2024-05-15T12:42:11.856 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.860+0000 7f26f2c1b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:11.856 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.860+0000 7f26f2c1b640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:11.856 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.860+0000 7f26f2c1b640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:11.856 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.860+0000 7f26f241a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:11.856 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.860+0000 7f26f241a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:11.856 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.860+0000 7f26f241a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:11.856 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.861+0000 7f26f2c1b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:11.856 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.861+0000 7f26f2c1b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:11.856 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.861+0000 7f26f2c1b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:11.856 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.864+0000 7f26f241a640 10 monclient(hunting): handle_auth_done global_id 4186 payload 824 2024-05-15T12:42:11.856 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.864+0000 7f26f241a640 1 -- 172.21.15.117:0/3195787842 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f26ec142db0 msgr2=0x7f26ec135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:11.856 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.864+0000 7f26f241a640 1 --2- 172.21.15.117:0/3195787842 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f26ec142db0 0x7f26ec135810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:11.856 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.864+0000 7f26f241a640 1 -- 172.21.15.117:0/3195787842 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f26ec13f850 msgr2=0x7f26ec1352d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:11.856 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.864+0000 7f26f241a640 1 --2- 172.21.15.117:0/3195787842 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f26ec13f850 0x7f26ec1352d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:11.856 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.864+0000 7f26f241a640 10 monclient: _finish_hunting 0 2024-05-15T12:42:11.856 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.864+0000 7f26f241a640 1 monclient: found mon.noname-b 2024-05-15T12:42:11.857 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.864+0000 7f26f241a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:42:11.857 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.864+0000 7f26f241a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2024-05-15T12:42:11.857 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.864+0000 7f26f241a640 1 -- 172.21.15.117:0/3195787842 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26ec1442a0 con 0x7f26ec0b54b0 2024-05-15T12:42:11.858 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.868+0000 7f26f241a640 1 --2- 172.21.15.117:0/3195787842 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f26ec0b54b0 0x7f26ec143520 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f26e800c450 tx=0x7f26e800c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:42:11.858 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.868+0000 7f26e2ffd640 10 client.?.objecter ms_handle_connect 0x7f26ec0b54b0 2024-05-15T12:42:11.858 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.868+0000 7f26e2ffd640 10 client.?.objecter resend_mon_ops 2024-05-15T12:42:11.864 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.873+0000 7f26e2ffd640 1 -- 172.21.15.117:0/3195787842 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e80052d0 con 0x7f26ec0b54b0 2024-05-15T12:42:11.864 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.873+0000 7f26e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:11.864 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.873+0000 7f26e2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:42:11.864 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.873+0000 7f26e2ffd640 10 monclient: dump: 2024-05-15T12:42:11.864 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-05-15T12:42:11.864 INFO:teuthology.misc.health.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:11.864 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:11.864 INFO:teuthology.misc.health.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:11.864 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:11.864 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-05-15T12:42:11.864 INFO:teuthology.misc.health.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:11.864 INFO:teuthology.misc.health.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:11.864 INFO:teuthology.misc.health.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:11.864 INFO:teuthology.misc.health.smithi117.stderr: 2024-05-15T12:42:11.864 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.873+0000 7f26e2ffd640 10 monclient: _finish_auth 0 2024-05-15T12:42:11.865 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.873+0000 7f26e2ffd640 10 monclient: _check_auth_tickets 2024-05-15T12:42:11.865 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.873+0000 7f26e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:42:11.865 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.873+0000 7f26e2ffd640 1 -- 172.21.15.117:0/3195787842 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26e8005470 con 0x7f26ec0b54b0 2024-05-15T12:42:11.865 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.873+0000 7f26e2ffd640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:42:11.866 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.873+0000 7f26f0c17640 4 set_mon_vals no callback set 2024-05-15T12:42:11.866 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.875+0000 7f26e2ffd640 1 -- 172.21.15.117:0/3195787842 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e8012870 con 0x7f26ec0b54b0 2024-05-15T12:42:11.866 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.875+0000 7f26e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:11.866 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.875+0000 7f26e2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:42:11.866 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.875+0000 7f26e2ffd640 10 monclient: dump: 2024-05-15T12:42:11.866 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-05-15T12:42:11.866 INFO:teuthology.misc.health.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:11.866 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:11.866 INFO:teuthology.misc.health.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:11.866 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:11.866 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-05-15T12:42:11.866 INFO:teuthology.misc.health.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:11.866 INFO:teuthology.misc.health.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:11.866 INFO:teuthology.misc.health.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:11.867 INFO:teuthology.misc.health.smithi117.stderr: 2024-05-15T12:42:11.867 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.875+0000 7f26f467e640 5 monclient: authenticate success, global_id 4186 2024-05-15T12:42:11.867 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.875+0000 7f26f467e640 10 monclient: _renew_subs 2024-05-15T12:42:11.867 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.875+0000 7f26f467e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:42:11.867 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.875+0000 7f26f467e640 1 -- 172.21.15.117:0/3195787842 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26ec0b87b0 con 0x7f26ec0b54b0 2024-05-15T12:42:11.867 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.875+0000 7f26f467e640 10 client.4186.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:42:11.867 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.875+0000 7f26f467e640 10 monclient: _renew_subs 2024-05-15T12:42:11.867 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.875+0000 7f26f467e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:42:11.867 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.875+0000 7f26f467e640 1 -- 172.21.15.117:0/3195787842 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26ec0b8a30 con 0x7f26ec0b54b0 2024-05-15T12:42:11.867 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.875+0000 7f26f467e640 1 librados: init done 2024-05-15T12:42:11.868 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.876+0000 7f26e0ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:42:11.868 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.876+0000 7f26e0ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:42:11.868 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.876+0000 7f26e0ff9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:42:11.868 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.876+0000 7f26e0ff9640 1 -- 172.21.15.117:0/3195787842 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26ec135e10 con 0x7f26ec0b54b0 2024-05-15T12:42:11.906 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.916+0000 7f26e2ffd640 1 -- 172.21.15.117:0/3195787842 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 3) v1 ==== 89616+0+0 (secure 0 0 0) 0x7f26e8012a10 con 0x7f26ec0b54b0 2024-05-15T12:42:11.907 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.917+0000 7f26e2ffd640 1 --2- 172.21.15.117:0/3195787842 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f26d006cd30 0x7f26d006f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:11.907 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.917+0000 7f26f1c19640 1 --2- 172.21.15.117:0/3195787842 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f26d006cd30 0x7f26d006f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:11.907 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.917+0000 7f26f1c19640 10 monclient: get_auth_request con 0x7f26d006cd30 auth_method 0 2024-05-15T12:42:11.908 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.918+0000 7f26e2ffd640 10 client.4186.objecter ms_handle_connect 0x7f26d006cd30 2024-05-15T12:42:11.908 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.918+0000 7f26f1c19640 1 --2- 172.21.15.117:0/3195787842 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f26d006cd30 0x7f26d006f1f0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f26dc036000 tx=0x7f26dc030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:42:11.911 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.921+0000 7f26e2ffd640 1 -- 172.21.15.117:0/3195787842 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f26e8057b50 con 0x7f26ec0b54b0 2024-05-15T12:42:11.911 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.921+0000 7f26e2ffd640 10 client.4186.objecter ms_dispatch 0x7f26ec000b90 osd_map(10..10 src has 1..10) v4 2024-05-15T12:42:11.911 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.921+0000 7f26e2ffd640 3 client.4186.objecter handle_osd_map got epochs [10,10] > 0 2024-05-15T12:42:11.911 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.921+0000 7f26e2ffd640 3 client.4186.objecter handle_osd_map decoding full epoch 10 2024-05-15T12:42:11.911 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:11.921+0000 7f26e2ffd640 20 client.4186.objecter dump_active .. 0 homeless 2024-05-15T12:42:12.364 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:12.374+0000 7f26e2ffd640 1 -- 172.21.15.117:0/3195787842 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f26ec0b87b0 con 0x7f26ec0b54b0 2024-05-15T12:42:12.364 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:12.374+0000 7f26e2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:42:12.364 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:12.374+0000 7f26e2ffd640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:42:12.732 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:12.742+0000 7f26e0ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:42:12.732 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:12.742+0000 7f26e0ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:42:12.732 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:12.742+0000 7f26e0ff9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:42:12.733 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:12.742+0000 7f26e0ff9640 1 -- 172.21.15.117:0/3195787842 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f26ec1447f0 con 0x7f26ec0b54b0 2024-05-15T12:42:13.355 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:13.365+0000 7f26e2ffd640 1 -- 172.21.15.117:0/3195787842 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mgrmap(e 4) v1 ==== 89722+0+0 (secure 0 0 0) 0x7f26e8012d00 con 0x7f26ec0b54b0 2024-05-15T12:42:13.361 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:13.371+0000 7f26e2ffd640 1 -- 172.21.15.117:0/3195787842 <== mon.1 v2:172.21.15.178:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f26ec1447f0 con 0x7f26ec0b54b0 2024-05-15T12:42:13.361 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:13.371+0000 7f26e2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:42:13.362 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:13.371+0000 7f26e2ffd640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:42:13.362 INFO:teuthology.misc.health.smithi117.stdout: 2024-05-15T12:42:13.362 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"8817f6cf-d933-4825-b972-575ef59f5ba9","created":"2024-05-15T12:42:04.264009+0000","modified":"2024-05-15T12:42:09.879357+0000","last_up_change":"0.000000","last_in_change":"2024-05-15T12:42:09.879357+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"aaffd6d9-1a00-4793-9d1f-65a30501bec4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"efd4e8ed-699c-4ccb-b989-cfadc7aaed35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"708e4e3e-7e79-480f-9dbf-7b972c2f2811","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e7b5d7d7-515c-42d9-9c41-b1ca3977865e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa647c2c-942b-4be3-afb9-6dba6b0a8cf3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5f9bc37-4bd6-4643-8f0d-e53c5f2c8782","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d5af075d-3eaa-4cb5-b6e9-6f01c201f2ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"323233c7-a3af-4fb0-ac70-de523a279b44","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-15T12:42:13.364 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:13.374+0000 7f26e0ff9640 10 librados: watch_flush enter 2024-05-15T12:42:13.364 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:13.374+0000 7f26e0ff9640 10 librados: watch_flush exit 2024-05-15T12:42:13.365 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:13.375+0000 7f26e0ff9640 20 client.4186.objecter shutdown clearing up homeless session... 2024-05-15T12:42:13.365 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:13.375+0000 7f26e0ff9640 10 client.4186.objecter successfully canceled tick 2024-05-15T12:42:13.365 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:13.375+0000 7f26e0ff9640 1 -- 172.21.15.117:0/3195787842 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f26d006cd30 msgr2=0x7f26d006f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:13.366 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:13.375+0000 7f26e0ff9640 1 --2- 172.21.15.117:0/3195787842 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f26d006cd30 0x7f26d006f1f0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f26dc036000 tx=0x7f26dc030000 comp rx=0 tx=0).stop 2024-05-15T12:42:13.366 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:13.375+0000 7f26e0ff9640 10 monclient: shutdown 2024-05-15T12:42:13.366 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:13.375+0000 7f26e0ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:42:13.366 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:13.375+0000 7f26e0ff9640 1 -- 172.21.15.117:0/3195787842 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f26ec0b54b0 msgr2=0x7f26ec143520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:13.366 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:13.375+0000 7f26e0ff9640 1 --2- 172.21.15.117:0/3195787842 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f26ec0b54b0 0x7f26ec143520 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f26e800c450 tx=0x7f26e800c910 comp rx=0 tx=0).stop 2024-05-15T12:42:13.366 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:13.375+0000 7f26e0ff9640 1 -- 172.21.15.117:0/3195787842 shutdown_connections 2024-05-15T12:42:13.366 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:13.375+0000 7f26e0ff9640 1 --2- 172.21.15.117:0/3195787842 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f26ec142db0 0x7f26ec135810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:13.366 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:13.375+0000 7f26e0ff9640 1 --2- 172.21.15.117:0/3195787842 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f26ec13f850 0x7f26ec1352d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:13.366 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:13.375+0000 7f26e0ff9640 1 --2- 172.21.15.117:0/3195787842 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f26d006cd30 0x7f26d006f1f0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:13.366 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:13.375+0000 7f26e0ff9640 1 --2- 172.21.15.117:0/3195787842 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f26ec0b54b0 0x7f26ec143520 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:13.366 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:13.375+0000 7f26e0ff9640 1 -- 172.21.15.117:0/3195787842 >> 172.21.15.117:0/3195787842 conn(0x7f26ec06d370 msgr2=0x7f26ec06d750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:42:13.366 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:13.375+0000 7f26e0ff9640 1 -- 172.21.15.117:0/3195787842 shutdown_connections 2024-05-15T12:42:13.366 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:13.375+0000 7f26e0ff9640 1 -- 172.21.15.117:0/3195787842 wait complete. 2024-05-15T12:42:13.366 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:13.375+0000 7f26e0ff9640 1 librados: shutdown 2024-05-15T12:42:13.382 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-05-15T12:42:17.718 INFO:tasks.ceph.osd.6.smithi178.stderr:==00:00:00:07.338 34487== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:17.755 INFO:tasks.ceph.osd.7.smithi178.stderr:==00:00:00:07.374 34488== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:17.787 INFO:tasks.ceph.osd.6.smithi178.stderr:==00:00:00:07.407 34487== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:17.817 INFO:tasks.ceph.osd.7.smithi178.stderr:==00:00:00:07.436 34488== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:17.944 INFO:tasks.ceph.osd.5.smithi178.stderr:==00:00:00:07.567 34486== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:17.949 INFO:tasks.ceph.osd.4.smithi178.stderr:==00:00:00:07.576 34485== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:17.963 INFO:tasks.ceph.osd.2.smithi117.stderr:==00:00:00:07.571 34653== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:17.975 INFO:tasks.ceph.osd.3.smithi117.stderr:==00:00:00:07.593 34651== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:18.003 INFO:tasks.ceph.osd.5.smithi178.stderr:==00:00:00:07.626 34486== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:18.013 INFO:tasks.ceph.osd.4.smithi178.stderr:==00:00:00:07.640 34485== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:18.031 INFO:tasks.ceph.osd.2.smithi117.stderr:==00:00:00:07.639 34653== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:18.053 INFO:tasks.ceph.osd.3.smithi117.stderr:==00:00:00:07.670 34651== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:18.328 INFO:tasks.ceph.osd.1.smithi117.stderr:==00:00:00:07.937 34652== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:18.333 INFO:tasks.ceph.osd.6.smithi178.stderr:2024-05-15T12:42:18.338+0000 5dce500 -1 Falling back to public interface 2024-05-15T12:42:18.352 INFO:tasks.ceph.osd.0.smithi117.stderr:==00:00:00:07.973 34650== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:18.363 INFO:tasks.ceph.osd.7.smithi178.stderr:2024-05-15T12:42:18.362+0000 5dce500 -1 Falling back to public interface 2024-05-15T12:42:18.393 INFO:tasks.ceph.osd.1.smithi117.stderr:==00:00:00:08.002 34652== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:18.412 INFO:tasks.ceph.osd.0.smithi117.stderr:==00:00:00:08.034 34650== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:18.547 INFO:tasks.ceph.osd.5.smithi178.stderr:2024-05-15T12:42:18.551+0000 5dce500 -1 Falling back to public interface 2024-05-15T12:42:18.590 INFO:tasks.ceph.osd.4.smithi178.stderr:2024-05-15T12:42:18.593+0000 5dce500 -1 Falling back to public interface 2024-05-15T12:42:18.609 INFO:tasks.ceph.osd.3.smithi117.stderr:2024-05-15T12:42:18.612+0000 5dce500 -1 Falling back to public interface 2024-05-15T12:42:18.612 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-05-15T12:42:18.616+0000 5dce500 -1 Falling back to public interface 2024-05-15T12:42:18.882 INFO:tasks.ceph.osd.6.smithi178.stderr:==00:00:00:08.502 34487== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:18.890 INFO:tasks.ceph.osd.7.smithi178.stderr:==00:00:00:08.509 34488== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:18.935 INFO:tasks.ceph.osd.1.smithi117.stderr:2024-05-15T12:42:18.940+0000 5dce500 -1 Falling back to public interface 2024-05-15T12:42:18.973 INFO:tasks.ceph.osd.0.smithi117.stderr:2024-05-15T12:42:18.978+0000 5dce500 -1 Falling back to public interface 2024-05-15T12:42:19.053 INFO:tasks.ceph.osd.5.smithi178.stderr:==00:00:00:08.676 34486== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:19.106 INFO:tasks.ceph.osd.2.smithi117.stderr:==00:00:00:08.714 34653== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:19.109 INFO:tasks.ceph.osd.4.smithi178.stderr:==00:00:00:08.737 34485== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:19.118 INFO:tasks.ceph.osd.3.smithi117.stderr:==00:00:00:08.735 34651== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:19.248 INFO:tasks.ceph.osd.6.smithi178.stderr:==00:00:00:08.868 34487== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:19.262 INFO:tasks.ceph.osd.7.smithi178.stderr:==00:00:00:08.882 34488== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:19.420 INFO:tasks.ceph.osd.5.smithi178.stderr:==00:00:00:09.043 34486== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:19.455 INFO:tasks.ceph.osd.1.smithi117.stderr:==00:00:00:09.064 34652== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:19.474 INFO:tasks.ceph.osd.2.smithi117.stderr:==00:00:00:09.083 34653== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:19.481 INFO:tasks.ceph.osd.4.smithi178.stderr:==00:00:00:09.109 34485== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:19.493 INFO:tasks.ceph.osd.0.smithi117.stderr:==00:00:00:09.115 34650== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:19.516 INFO:tasks.ceph.osd.3.smithi117.stderr:==00:00:00:09.134 34651== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:19.574 INFO:tasks.ceph.osd.7.smithi178.stderr:==00:00:00:09.194 34488== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:19.585 INFO:tasks.ceph.osd.6.smithi178.stderr:==00:00:00:09.205 34487== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:19.734 INFO:tasks.ceph.osd.5.smithi178.stderr:==00:00:00:09.357 34486== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:19.820 INFO:tasks.ceph.osd.1.smithi117.stderr:==00:00:00:09.430 34652== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:19.838 INFO:tasks.ceph.osd.3.smithi117.stderr:==00:00:00:09.455 34651== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:19.843 INFO:tasks.ceph.osd.7.smithi178.stderr:==00:00:00:09.463 34488== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:19.846 INFO:tasks.ceph.osd.4.smithi178.stderr:==00:00:00:09.474 34485== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:19.847 INFO:tasks.ceph.osd.2.smithi117.stderr:==00:00:00:09.455 34653== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:19.873 INFO:tasks.ceph.osd.0.smithi117.stderr:==00:00:00:09.494 34650== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:19.876 INFO:tasks.ceph.osd.6.smithi178.stderr:==00:00:00:09.496 34487== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:20.005 INFO:tasks.ceph.osd.5.smithi178.stderr:==00:00:00:09.628 34486== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:20.103 INFO:tasks.ceph.osd.3.smithi117.stderr:==00:00:00:09.721 34651== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:20.107 INFO:tasks.ceph.osd.7.smithi178.stderr:==00:00:00:09.727 34488== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:20.136 INFO:tasks.ceph.osd.4.smithi178.stderr:==00:00:00:09.764 34485== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:20.143 INFO:tasks.ceph.osd.2.smithi117.stderr:==00:00:00:09.751 34653== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:20.165 INFO:tasks.ceph.osd.6.smithi178.stderr:==00:00:00:09.784 34487== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:20.179 INFO:tasks.ceph.osd.1.smithi117.stderr:==00:00:00:09.788 34652== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:20.188 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-15T12:42:20.246 INFO:tasks.ceph.osd.0.smithi117.stderr:==00:00:00:09.868 34650== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:20.277 INFO:tasks.ceph.osd.5.smithi178.stderr:==00:00:00:09.900 34486== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:20.282 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.292+0000 7ff7115de640 10 monclient: get_monmap_and_config 2024-05-15T12:42:20.282 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.292+0000 7ff7115de640 10 monclient: build_initial_monmap 2024-05-15T12:42:20.282 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.292+0000 7ff7115de640 10 monclient: monmap: 2024-05-15T12:42:20.282 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-05-15T12:42:20.282 INFO:teuthology.misc.health.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:20.282 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-05-15T12:42:20.293493+0000 2024-05-15T12:42:20.282 INFO:teuthology.misc.health.smithi117.stderr:created 2024-05-15T12:42:20.293493+0000 2024-05-15T12:42:20.282 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:42:20.283 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-05-15T12:42:20.283 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T12:42:20.283 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T12:42:20.283 INFO:teuthology.misc.health.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T12:42:20.283 INFO:teuthology.misc.health.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T12:42:20.283 INFO:teuthology.misc.health.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T12:42:20.283 INFO:teuthology.misc.health.smithi117.stderr: 2024-05-15T12:42:20.285 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.293+0000 7ff7115de640 1 Processor -- start 2024-05-15T12:42:20.285 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.293+0000 7ff7115de640 1 -- start start 2024-05-15T12:42:20.286 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.293+0000 7ff7115de640 10 monclient: init 2024-05-15T12:42:20.286 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.293+0000 7ff7115de640 10 monclient: _reopen_session rank -1 2024-05-15T12:42:20.286 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.293+0000 7ff7115de640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-05-15T12:42:20.286 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.293+0000 7ff7115de640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7ff70c1431e0 0x7ff70c0b5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:20.286 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.293+0000 7ff7115de640 10 monclient(hunting): picked mon.noname-c con 0x7ff70c1431e0 addr v2:172.21.15.178:3300/0 2024-05-15T12:42:20.286 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.293+0000 7ff7115de640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff70c0b1d10 0x7ff70c0b2170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:20.286 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.293+0000 7ff7115de640 10 monclient(hunting): picked mon.noname-b con 0x7ff70c0b1d10 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:42:20.286 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.293+0000 7ff7115de640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7ff70c0afd80 0x7ff70c0b0210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:20.286 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.293+0000 7ff7115de640 10 monclient(hunting): picked mon.noname-a con 0x7ff70c0afd80 addr v2:172.21.15.117:3300/0 2024-05-15T12:42:20.286 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.293+0000 7ff7115de640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:20.286 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.293+0000 7ff7115de640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff70c0b2840 con 0x7ff70c0afd80 2024-05-15T12:42:20.286 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.293+0000 7ff7115de640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:20.286 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.293+0000 7ff7115de640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff70c0b0750 con 0x7ff70c1431e0 2024-05-15T12:42:20.287 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.293+0000 7ff7115de640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:20.287 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.293+0000 7ff7115de640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff70c0b18f0 con 0x7ff70c0b1d10 2024-05-15T12:42:20.287 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.293+0000 7ff7115de640 10 monclient(hunting): _renew_subs 2024-05-15T12:42:20.287 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.293+0000 7ff7115de640 10 monclient(hunting): authenticate will time out at 1303.535710s 2024-05-15T12:42:20.287 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.294+0000 7ff703fff640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff70c0b1d10 0x7ff70c0b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:20.287 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.294+0000 7ff70b7fe640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7ff70c0afd80 0x7ff70c0b0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:20.287 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.294+0000 7ff70affd640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7ff70c1431e0 0x7ff70c0b5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:20.287 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.294+0000 7ff703fff640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff70c0b1d10 0x7ff70c0b2170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.117:48994/0 (socket says 172.21.15.117:48994) 2024-05-15T12:42:20.287 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.294+0000 7ff70affd640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7ff70c1431e0 0x7ff70c0b5220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:36788/0 (socket says 172.21.15.117:36788) 2024-05-15T12:42:20.287 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.294+0000 7ff703fff640 1 -- 172.21.15.117:0/1868661325 learned_addr learned my addr 172.21.15.117:0/1868661325 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:42:20.287 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.294+0000 7ff703fff640 10 monclient(hunting): get_auth_request con 0x7ff70c0b1d10 auth_method 0 2024-05-15T12:42:20.287 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.294+0000 7ff703fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:20.287 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.294+0000 7ff703fff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:20.287 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.294+0000 7ff703fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:20.287 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.294+0000 7ff70affd640 10 monclient(hunting): get_auth_request con 0x7ff70c1431e0 auth_method 0 2024-05-15T12:42:20.288 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.294+0000 7ff70affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:20.288 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.294+0000 7ff70affd640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:20.288 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.294+0000 7ff70affd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:20.288 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.294+0000 7ff70b7fe640 10 monclient(hunting): get_auth_request con 0x7ff70c0afd80 auth_method 0 2024-05-15T12:42:20.288 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.294+0000 7ff70b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:20.288 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.294+0000 7ff70b7fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:20.288 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.294+0000 7ff70b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:20.288 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.295+0000 7ff70affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:20.288 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.295+0000 7ff70affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:20.288 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.295+0000 7ff70affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:20.288 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.295+0000 7ff703fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:20.288 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.295+0000 7ff703fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:20.288 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.295+0000 7ff703fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:20.288 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.295+0000 7ff70b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:20.288 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.295+0000 7ff70b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:20.289 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.295+0000 7ff70b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:20.289 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.296+0000 7ff70affd640 10 monclient(hunting): handle_auth_done global_id 4213 payload 274 2024-05-15T12:42:20.289 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.296+0000 7ff70affd640 1 -- 172.21.15.117:0/1868661325 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff70c0b1d10 msgr2=0x7ff70c0b2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:42:20.289 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.296+0000 7ff70affd640 1 --2- 172.21.15.117:0/1868661325 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff70c0b1d10 0x7ff70c0b2170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:20.289 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.296+0000 7ff70affd640 1 -- 172.21.15.117:0/1868661325 >> v2:172.21.15.117:3300/0 conn(0x7ff70c0afd80 msgr2=0x7ff70c0b0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:42:20.290 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.296+0000 7ff70affd640 1 --2- 172.21.15.117:0/1868661325 >> v2:172.21.15.117:3300/0 conn(0x7ff70c0afd80 0x7ff70c0b0210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:20.290 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.296+0000 7ff70affd640 10 monclient: _finish_hunting 0 2024-05-15T12:42:20.290 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.296+0000 7ff70affd640 1 monclient: found mon.noname-c 2024-05-15T12:42:20.290 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.296+0000 7ff70affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:42:20.290 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.296+0000 7ff70affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.178:3300/0 2024-05-15T12:42:20.290 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.296+0000 7ff70affd640 1 -- 172.21.15.117:0/1868661325 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff70c0b5760 con 0x7ff70c1431e0 2024-05-15T12:42:20.290 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.298+0000 7ff70affd640 1 --2- 172.21.15.117:0/1868661325 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff70c1431e0 0x7ff70c0b5220 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff70c06dee0 tx=0x7ff6fc02b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd496fdceb7bcff2 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:42:20.292 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.302+0000 7ff70a7fc640 1 -- 172.21.15.117:0/1868661325 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6fc038070 con 0x7ff70c1431e0 2024-05-15T12:42:20.292 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.302+0000 7ff70a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:20.292 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.302+0000 7ff70a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:42:20.292 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.302+0000 7ff70a7fc640 10 monclient: dump: 2024-05-15T12:42:20.292 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-05-15T12:42:20.292 INFO:teuthology.misc.health.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:20.292 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:20.292 INFO:teuthology.misc.health.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:20.292 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:20.293 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-05-15T12:42:20.293 INFO:teuthology.misc.health.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:20.293 INFO:teuthology.misc.health.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:20.293 INFO:teuthology.misc.health.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:20.293 INFO:teuthology.misc.health.smithi117.stderr: 2024-05-15T12:42:20.293 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.302+0000 7ff70a7fc640 10 monclient: _finish_auth 0 2024-05-15T12:42:20.293 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.302+0000 7ff70a7fc640 10 monclient: _check_auth_tickets 2024-05-15T12:42:20.293 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.302+0000 7ff70a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:42:20.293 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.302+0000 7ff70a7fc640 1 -- 172.21.15.117:0/1868661325 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6fc031700 con 0x7ff70c1431e0 2024-05-15T12:42:20.293 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.302+0000 7ff70a7fc640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:42:20.293 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.302+0000 7ff70a7fc640 1 -- 172.21.15.117:0/1868661325 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6fc031ad0 con 0x7ff70c1431e0 2024-05-15T12:42:20.293 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.302+0000 7ff70a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:20.293 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.302+0000 7ff70a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:42:20.293 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.302+0000 7ff70a7fc640 10 monclient: dump: 2024-05-15T12:42:20.293 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-05-15T12:42:20.294 INFO:teuthology.misc.health.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:20.294 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:20.294 INFO:teuthology.misc.health.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:20.294 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:20.294 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-05-15T12:42:20.294 INFO:teuthology.misc.health.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:20.294 INFO:teuthology.misc.health.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:20.294 INFO:teuthology.misc.health.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:20.294 INFO:teuthology.misc.health.smithi117.stderr: 2024-05-15T12:42:20.294 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.302+0000 7ff7115de640 5 monclient: authenticate success, global_id 4213 2024-05-15T12:42:20.294 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.302+0000 7ff7115de640 10 monclient: get_monmap_and_config success 2024-05-15T12:42:20.294 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.302+0000 7ff7115de640 4 set_mon_vals no callback set 2024-05-15T12:42:20.298 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.303+0000 7ff7115de640 10 monclient: shutdown 2024-05-15T12:42:20.298 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.303+0000 7ff7115de640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:42:20.298 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.303+0000 7ff7115de640 1 -- 172.21.15.117:0/1868661325 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff70c1431e0 msgr2=0x7ff70c0b5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:20.299 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.303+0000 7ff7115de640 1 --2- 172.21.15.117:0/1868661325 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff70c1431e0 0x7ff70c0b5220 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff70c06dee0 tx=0x7ff6fc02b7a0 comp rx=0 tx=0).stop 2024-05-15T12:42:20.299 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.303+0000 7ff7115de640 1 -- 172.21.15.117:0/1868661325 shutdown_connections 2024-05-15T12:42:20.299 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.303+0000 7ff7115de640 1 --2- 172.21.15.117:0/1868661325 >> v2:172.21.15.117:3300/0 conn(0x7ff70c0afd80 0x7ff70c0b0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:20.299 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.303+0000 7ff7115de640 1 --2- 172.21.15.117:0/1868661325 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff70c0b1d10 0x7ff70c0b2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:20.299 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.303+0000 7ff7115de640 1 --2- 172.21.15.117:0/1868661325 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff70c1431e0 0x7ff70c0b5220 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:20.299 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.303+0000 7ff7115de640 1 -- 172.21.15.117:0/1868661325 >> 172.21.15.117:0/1868661325 conn(0x7ff70c06d3a0 msgr2=0x7ff70c06d7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:42:20.299 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.303+0000 7ff7115de640 1 -- 172.21.15.117:0/1868661325 shutdown_connections 2024-05-15T12:42:20.299 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.303+0000 7ff7115de640 1 -- 172.21.15.117:0/1868661325 wait complete. 2024-05-15T12:42:20.299 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.303+0000 7ff7115de640 10 monclient: build_initial_monmap 2024-05-15T12:42:20.299 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.303+0000 7ff7115de640 10 monclient: monmap: 2024-05-15T12:42:20.299 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-05-15T12:42:20.299 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:42:20.299 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-05-15T12:42:20.299 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-05-15T12:42:20.299 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:42:20.299 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-05-15T12:42:20.300 INFO:teuthology.misc.health.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T12:42:20.300 INFO:teuthology.misc.health.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T12:42:20.300 INFO:teuthology.misc.health.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T12:42:20.300 INFO:teuthology.misc.health.smithi117.stderr: 2024-05-15T12:42:20.300 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.304+0000 7ff7115de640 1 librados: starting msgr at 2024-05-15T12:42:20.300 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.304+0000 7ff7115de640 1 librados: starting objecter 2024-05-15T12:42:20.300 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.304+0000 7ff7115de640 1 Processor -- start 2024-05-15T12:42:20.300 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.304+0000 7ff7115de640 1 -- start start 2024-05-15T12:42:20.300 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.304+0000 7ff7115de640 1 librados: setting wanted keys 2024-05-15T12:42:20.300 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.304+0000 7ff7115de640 1 librados: calling monclient init 2024-05-15T12:42:20.300 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.304+0000 7ff7115de640 10 monclient: init 2024-05-15T12:42:20.300 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.304+0000 7ff7115de640 10 monclient: _reopen_session rank -1 2024-05-15T12:42:20.300 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.304+0000 7ff7115de640 10 monclient: _add_conns ranks=[0,2,1] 2024-05-15T12:42:20.300 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.304+0000 7ff7115de640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff70c0afd80 0x7ff70c0a9c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:20.300 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.304+0000 7ff7115de640 10 monclient(hunting): picked mon.noname-a con 0x7ff70c0afd80 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T12:42:20.300 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.304+0000 7ff7115de640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff70c0b1d10 0x7ff70c0aa170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:20.300 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.304+0000 7ff7115de640 10 monclient(hunting): picked mon.noname-c con 0x7ff70c0b1d10 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:42:20.301 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.304+0000 7ff7115de640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff70c1431e0 0x7ff70c1d7e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:20.301 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.304+0000 7ff7115de640 10 monclient(hunting): picked mon.noname-b con 0x7ff70c1431e0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:42:20.301 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.304+0000 7ff7115de640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:20.301 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.304+0000 7ff7115de640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff70c0b2b50 con 0x7ff70c0afd80 2024-05-15T12:42:20.301 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.304+0000 7ff7115de640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:20.301 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.304+0000 7ff7115de640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff70c1435c0 con 0x7ff70c1431e0 2024-05-15T12:42:20.301 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.304+0000 7ff7115de640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:20.301 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.304+0000 7ff7115de640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff70c0b2cd0 con 0x7ff70c0b1d10 2024-05-15T12:42:20.301 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.304+0000 7ff7115de640 10 monclient(hunting): _renew_subs 2024-05-15T12:42:20.301 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.304+0000 7ff7115de640 10 monclient(hunting): authenticate will time out at 1303.546615s 2024-05-15T12:42:20.301 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.305+0000 7ff703fff640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff70c0b1d10 0x7ff70c0aa170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:20.301 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.305+0000 7ff70affd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff70c0afd80 0x7ff70c0a9c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:20.301 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.305+0000 7ff70b7fe640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff70c1431e0 0x7ff70c1d7e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:20.301 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.305+0000 7ff703fff640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff70c0b1d10 0x7ff70c0aa170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.117:49006/0 (socket says 172.21.15.117:49006) 2024-05-15T12:42:20.301 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.305+0000 7ff703fff640 1 -- 172.21.15.117:0/83348258 learned_addr learned my addr 172.21.15.117:0/83348258 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:42:20.301 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.305+0000 7ff703fff640 10 monclient(hunting): get_auth_request con 0x7ff70c0b1d10 auth_method 0 2024-05-15T12:42:20.302 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.305+0000 7ff703fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:20.302 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.305+0000 7ff703fff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:20.302 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.305+0000 7ff703fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:20.302 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.305+0000 7ff70b7fe640 10 monclient(hunting): get_auth_request con 0x7ff70c1431e0 auth_method 0 2024-05-15T12:42:20.302 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.305+0000 7ff70b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:20.302 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.305+0000 7ff70b7fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:20.302 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.305+0000 7ff70b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:20.302 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.305+0000 7ff70affd640 10 monclient(hunting): get_auth_request con 0x7ff70c0afd80 auth_method 0 2024-05-15T12:42:20.302 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.305+0000 7ff70affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:20.302 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.305+0000 7ff70affd640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:20.302 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.305+0000 7ff70affd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:20.302 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.306+0000 7ff703fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:20.302 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.306+0000 7ff703fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:20.302 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.306+0000 7ff703fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:20.302 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.306+0000 7ff70b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:20.303 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.306+0000 7ff70b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:20.303 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.306+0000 7ff70b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:20.303 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.306+0000 7ff70affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:20.303 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.306+0000 7ff70affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:20.303 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.306+0000 7ff70affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:20.303 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.308+0000 7ff70b7fe640 10 monclient(hunting): handle_auth_done global_id 4216 payload 824 2024-05-15T12:42:20.303 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.308+0000 7ff70b7fe640 1 -- 172.21.15.117:0/83348258 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff70c0b1d10 msgr2=0x7ff70c0aa170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:20.303 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.308+0000 7ff70b7fe640 1 --2- 172.21.15.117:0/83348258 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff70c0b1d10 0x7ff70c0aa170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:20.303 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.308+0000 7ff70b7fe640 1 -- 172.21.15.117:0/83348258 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff70c0afd80 msgr2=0x7ff70c0a9c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:20.303 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.308+0000 7ff70b7fe640 1 --2- 172.21.15.117:0/83348258 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff70c0afd80 0x7ff70c0a9c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:20.303 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.308+0000 7ff70b7fe640 10 monclient: _finish_hunting 0 2024-05-15T12:42:20.303 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.308+0000 7ff70b7fe640 1 monclient: found mon.noname-b 2024-05-15T12:42:20.303 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.308+0000 7ff70b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:42:20.303 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.308+0000 7ff70b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2024-05-15T12:42:20.303 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.308+0000 7ff70b7fe640 1 -- 172.21.15.117:0/83348258 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff70c0ab550 con 0x7ff70c1431e0 2024-05-15T12:42:20.305 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.310+0000 7ff70b7fe640 1 --2- 172.21.15.117:0/83348258 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff70c1431e0 0x7ff70c1d7e90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff70400ef00 tx=0x7ff70400b480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:42:20.305 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.310+0000 7ff7037fe640 10 client.?.objecter ms_handle_connect 0x7ff70c1431e0 2024-05-15T12:42:20.305 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.310+0000 7ff7037fe640 10 client.?.objecter resend_mon_ops 2024-05-15T12:42:20.305 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.314+0000 7ff7037fe640 1 -- 172.21.15.117:0/83348258 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff704018020 con 0x7ff70c1431e0 2024-05-15T12:42:20.305 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.314+0000 7ff7037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:20.305 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.314+0000 7ff7037fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:42:20.305 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.314+0000 7ff7037fe640 10 monclient: dump: 2024-05-15T12:42:20.305 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-05-15T12:42:20.305 INFO:teuthology.misc.health.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:20.305 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:20.305 INFO:teuthology.misc.health.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:20.305 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:20.305 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-05-15T12:42:20.306 INFO:teuthology.misc.health.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:20.306 INFO:teuthology.misc.health.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:20.306 INFO:teuthology.misc.health.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:20.306 INFO:teuthology.misc.health.smithi117.stderr: 2024-05-15T12:42:20.306 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.314+0000 7ff7037fe640 10 monclient: _finish_auth 0 2024-05-15T12:42:20.306 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.314+0000 7ff7037fe640 10 monclient: _check_auth_tickets 2024-05-15T12:42:20.306 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.314+0000 7ff7037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:42:20.306 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.314+0000 7ff7037fe640 1 -- 172.21.15.117:0/83348258 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff704014400 con 0x7ff70c1431e0 2024-05-15T12:42:20.306 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.314+0000 7ff7037fe640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:42:20.306 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.314+0000 7ff7037fe640 1 -- 172.21.15.117:0/83348258 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff704009e60 con 0x7ff70c1431e0 2024-05-15T12:42:20.306 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.314+0000 7ff7115de640 5 monclient: authenticate success, global_id 4216 2024-05-15T12:42:20.306 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.314+0000 7ff7037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:20.306 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.314+0000 7ff7037fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:42:20.306 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.314+0000 7ff7037fe640 10 monclient: dump: 2024-05-15T12:42:20.306 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-05-15T12:42:20.307 INFO:teuthology.misc.health.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:20.307 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:20.307 INFO:teuthology.misc.health.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:20.307 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:20.307 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-05-15T12:42:20.307 INFO:teuthology.misc.health.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:20.307 INFO:teuthology.misc.health.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:20.307 INFO:teuthology.misc.health.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:20.307 INFO:teuthology.misc.health.smithi117.stderr: 2024-05-15T12:42:20.307 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.314+0000 7ff7115de640 10 monclient: _renew_subs 2024-05-15T12:42:20.307 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.314+0000 7ff709ffb640 4 set_mon_vals no callback set 2024-05-15T12:42:20.307 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.314+0000 7ff7115de640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:42:20.307 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.314+0000 7ff7115de640 1 -- 172.21.15.117:0/83348258 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff70c1cf800 con 0x7ff70c1431e0 2024-05-15T12:42:20.307 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.314+0000 7ff7115de640 10 client.4216.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:42:20.307 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.314+0000 7ff7115de640 10 monclient: _renew_subs 2024-05-15T12:42:20.308 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.314+0000 7ff7115de640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:42:20.308 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.314+0000 7ff7115de640 1 -- 172.21.15.117:0/83348258 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff70c0a94d0 con 0x7ff70c1431e0 2024-05-15T12:42:20.308 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.314+0000 7ff7115de640 1 librados: init done 2024-05-15T12:42:20.308 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.315+0000 7ff7115de640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:42:20.308 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.315+0000 7ff7115de640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:42:20.308 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.315+0000 7ff7115de640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:42:20.308 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.315+0000 7ff7115de640 1 -- 172.21.15.117:0/83348258 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff70c04f0e0 con 0x7ff70c1431e0 2024-05-15T12:42:20.322 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.332+0000 7ff7037fe640 1 -- 172.21.15.117:0/83348258 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 4) v1 ==== 89722+0+0 (secure 0 0 0) 0x7ff704005d10 con 0x7ff70c1431e0 2024-05-15T12:42:20.323 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.333+0000 7ff7037fe640 1 --2- 172.21.15.117:0/83348258 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7ff6ec06ce20 0x7ff6ec06f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:20.323 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.333+0000 7ff70affd640 1 --2- 172.21.15.117:0/83348258 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7ff6ec06ce20 0x7ff6ec06f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:20.323 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.333+0000 7ff70affd640 10 monclient: get_auth_request con 0x7ff6ec06ce20 auth_method 0 2024-05-15T12:42:20.323 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.333+0000 7ff7037fe640 10 client.4216.objecter ms_handle_connect 0x7ff6ec06ce20 2024-05-15T12:42:20.323 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.333+0000 7ff70affd640 1 --2- 172.21.15.117:0/83348258 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7ff6ec06ce20 0x7ff6ec06f2e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff6fc035000 tx=0x7ff6fc008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:42:20.325 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.335+0000 7ff7037fe640 1 -- 172.21.15.117:0/83348258 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7ff704058450 con 0x7ff70c1431e0 2024-05-15T12:42:20.325 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.335+0000 7ff7037fe640 10 client.4216.objecter ms_dispatch 0x7ff70c000b90 osd_map(10..10 src has 1..10) v4 2024-05-15T12:42:20.325 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.335+0000 7ff7037fe640 3 client.4216.objecter handle_osd_map got epochs [10,10] > 0 2024-05-15T12:42:20.325 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.335+0000 7ff7037fe640 3 client.4216.objecter handle_osd_map decoding full epoch 10 2024-05-15T12:42:20.325 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.335+0000 7ff7037fe640 20 client.4216.objecter dump_active .. 0 homeless 2024-05-15T12:42:20.371 INFO:tasks.ceph.osd.3.smithi117.stderr:==00:00:00:09.989 34651== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:20.382 INFO:tasks.ceph.osd.7.smithi178.stderr:==00:00:00:10.002 34488== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:20.432 INFO:tasks.ceph.osd.2.smithi117.stderr:==00:00:00:10.040 34653== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:20.432 INFO:tasks.ceph.osd.4.smithi178.stderr:==00:00:00:10.060 34485== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:20.455 INFO:tasks.ceph.osd.6.smithi178.stderr:==00:00:00:10.075 34487== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:20.474 INFO:tasks.ceph.osd.1.smithi117.stderr:==00:00:00:10.083 34652== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:20.539 INFO:tasks.ceph.osd.0.smithi117.stderr:==00:00:00:10.160 34650== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:20.541 INFO:tasks.ceph.osd.5.smithi178.stderr:==00:00:00:10.164 34486== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:20.566 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.576+0000 7ff7037fe640 1 -- 172.21.15.117:0/83348258 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7ff704022030 con 0x7ff70c1431e0 2024-05-15T12:42:20.566 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.576+0000 7ff7037fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:42:20.566 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.576+0000 7ff7037fe640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:42:20.639 INFO:tasks.ceph.osd.3.smithi117.stderr:==00:00:00:10.257 34651== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:20.650 INFO:tasks.ceph.osd.7.smithi178.stderr:==00:00:00:10.270 34488== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:20.720 INFO:tasks.ceph.osd.2.smithi117.stderr:==00:00:00:10.328 34653== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:20.720 INFO:tasks.ceph.osd.4.smithi178.stderr:==00:00:00:10.348 34485== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:20.741 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.751+0000 7ff7115de640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:42:20.741 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.751+0000 7ff7115de640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:42:20.741 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.751+0000 7ff7115de640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:42:20.741 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.751+0000 7ff7115de640 1 -- 172.21.15.117:0/83348258 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff70c0b64d0 con 0x7ff70c1431e0 2024-05-15T12:42:20.743 INFO:tasks.ceph.osd.6.smithi178.stderr:==00:00:00:10.363 34487== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:20.756 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.766+0000 7ff7037fe640 1 -- 172.21.15.117:0/83348258 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7ff7040597c0 con 0x7ff70c1431e0 2024-05-15T12:42:20.756 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.766+0000 7ff7037fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:42:20.757 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.766+0000 7ff7037fe640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:42:20.757 INFO:teuthology.misc.health.smithi117.stdout: 2024-05-15T12:42:20.757 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"8817f6cf-d933-4825-b972-575ef59f5ba9","created":"2024-05-15T12:42:04.264009+0000","modified":"2024-05-15T12:42:09.879357+0000","last_up_change":"0.000000","last_in_change":"2024-05-15T12:42:09.879357+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"aaffd6d9-1a00-4793-9d1f-65a30501bec4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"efd4e8ed-699c-4ccb-b989-cfadc7aaed35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"708e4e3e-7e79-480f-9dbf-7b972c2f2811","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e7b5d7d7-515c-42d9-9c41-b1ca3977865e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa647c2c-942b-4be3-afb9-6dba6b0a8cf3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5f9bc37-4bd6-4643-8f0d-e53c5f2c8782","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d5af075d-3eaa-4cb5-b6e9-6f01c201f2ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"323233c7-a3af-4fb0-ac70-de523a279b44","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-15T12:42:20.759 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.768+0000 7ff7115de640 10 librados: watch_flush enter 2024-05-15T12:42:20.759 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.768+0000 7ff7115de640 10 librados: watch_flush exit 2024-05-15T12:42:20.759 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.768+0000 7ff7115de640 20 client.4216.objecter shutdown clearing up homeless session... 2024-05-15T12:42:20.760 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.768+0000 7ff7115de640 10 client.4216.objecter successfully canceled tick 2024-05-15T12:42:20.760 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.768+0000 7ff7115de640 1 -- 172.21.15.117:0/83348258 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7ff6ec06ce20 msgr2=0x7ff6ec06f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:20.760 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.768+0000 7ff7115de640 1 --2- 172.21.15.117:0/83348258 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7ff6ec06ce20 0x7ff6ec06f2e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff6fc035000 tx=0x7ff6fc008030 comp rx=0 tx=0).stop 2024-05-15T12:42:20.760 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.768+0000 7ff7115de640 10 monclient: shutdown 2024-05-15T12:42:20.760 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.768+0000 7ff7115de640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:42:20.760 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.768+0000 7ff7115de640 1 -- 172.21.15.117:0/83348258 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff70c1431e0 msgr2=0x7ff70c1d7e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:20.760 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.768+0000 7ff7115de640 1 --2- 172.21.15.117:0/83348258 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff70c1431e0 0x7ff70c1d7e90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff70400ef00 tx=0x7ff70400b480 comp rx=0 tx=0).stop 2024-05-15T12:42:20.760 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.769+0000 7ff7115de640 1 -- 172.21.15.117:0/83348258 shutdown_connections 2024-05-15T12:42:20.760 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.769+0000 7ff7115de640 1 --2- 172.21.15.117:0/83348258 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7ff6ec06ce20 0x7ff6ec06f2e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:20.760 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.769+0000 7ff7115de640 1 --2- 172.21.15.117:0/83348258 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff70c1431e0 0x7ff70c1d7e90 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:20.760 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.769+0000 7ff7115de640 1 --2- 172.21.15.117:0/83348258 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff70c0b1d10 0x7ff70c0aa170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:20.760 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.769+0000 7ff7115de640 1 --2- 172.21.15.117:0/83348258 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff70c0afd80 0x7ff70c0a9c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:20.760 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.769+0000 7ff7115de640 1 -- 172.21.15.117:0/83348258 >> 172.21.15.117:0/83348258 conn(0x7ff70c06d3a0 msgr2=0x7ff70c074610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:42:20.760 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.769+0000 7ff7115de640 1 -- 172.21.15.117:0/83348258 shutdown_connections 2024-05-15T12:42:20.761 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.769+0000 7ff7115de640 1 -- 172.21.15.117:0/83348258 wait complete. 2024-05-15T12:42:20.761 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:20.769+0000 7ff7115de640 1 librados: shutdown 2024-05-15T12:42:20.770 INFO:tasks.ceph.osd.4.smithi178.stderr:==00:00:00:10.396 34485== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:20.771 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-05-15T12:42:20.772 INFO:tasks.ceph.osd.1.smithi117.stderr:==00:00:00:10.381 34652== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:20.808 INFO:tasks.ceph.osd.5.smithi178.stderr:==00:00:00:10.431 34486== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:20.827 INFO:tasks.ceph.osd.0.smithi117.stderr:==00:00:00:10.448 34650== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:20.903 INFO:tasks.ceph.osd.3.smithi117.stderr:==00:00:00:10.521 34651== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:20.914 INFO:tasks.ceph.osd.7.smithi178.stderr:==00:00:00:10.534 34488== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:21.008 INFO:tasks.ceph.osd.2.smithi117.stderr:==00:00:00:10.616 34653== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:21.041 INFO:tasks.ceph.osd.6.smithi178.stderr:==00:00:00:10.661 34487== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:21.060 INFO:tasks.ceph.osd.1.smithi117.stderr:==00:00:00:10.669 34652== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:21.073 INFO:tasks.ceph.osd.4.smithi178.stderr:==00:00:00:10.702 34485== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:21.081 INFO:tasks.ceph.osd.5.smithi178.stderr:==00:00:00:10.704 34486== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:21.119 INFO:tasks.ceph.osd.0.smithi117.stderr:==00:00:00:10.740 34650== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:21.171 INFO:tasks.ceph.osd.3.smithi117.stderr:==00:00:00:10.789 34651== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:21.186 INFO:tasks.ceph.osd.7.smithi178.stderr:==00:00:00:10.806 34488== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:21.296 INFO:tasks.ceph.osd.2.smithi117.stderr:==00:00:00:10.904 34653== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:21.329 INFO:tasks.ceph.osd.6.smithi178.stderr:==00:00:00:10.949 34487== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:21.348 INFO:tasks.ceph.osd.1.smithi117.stderr:==00:00:00:10.957 34652== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:21.350 INFO:tasks.ceph.osd.5.smithi178.stderr:==00:00:00:10.973 34486== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:21.366 INFO:tasks.ceph.osd.4.smithi178.stderr:==00:00:00:10.994 34485== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:21.406 INFO:tasks.ceph.osd.0.smithi117.stderr:==00:00:00:11.027 34650== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:21.436 INFO:tasks.ceph.osd.3.smithi117.stderr:==00:00:00:11.054 34651== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:21.588 INFO:tasks.ceph.osd.2.smithi117.stderr:==00:00:00:11.196 34653== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:21.638 INFO:tasks.ceph.osd.1.smithi117.stderr:==00:00:00:11.247 34652== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:21.689 INFO:tasks.ceph.osd.7.smithi178.stderr:==00:00:00:11.309 34488== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:21.706 INFO:tasks.ceph.osd.0.smithi117.stderr:==00:00:00:11.327 34650== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:21.756 INFO:tasks.ceph.osd.7.smithi178.stderr:==00:00:00:11.375 34488== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:21.855 INFO:tasks.ceph.osd.5.smithi178.stderr:==00:00:00:11.478 34486== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:21.868 INFO:tasks.ceph.osd.2.smithi117.stderr:==00:00:00:11.476 34653== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:21.881 INFO:tasks.ceph.osd.6.smithi178.stderr:==00:00:00:11.501 34487== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:21.904 INFO:tasks.ceph.osd.4.smithi178.stderr:==00:00:00:11.532 34485== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:21.913 INFO:tasks.ceph.osd.5.smithi178.stderr:==00:00:00:11.536 34486== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:21.915 INFO:tasks.ceph.osd.3.smithi117.stderr:==00:00:00:11.533 34651== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:21.922 INFO:tasks.ceph.osd.2.smithi117.stderr:==00:00:00:11.531 34653== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:21.931 INFO:tasks.ceph.osd.1.smithi117.stderr:==00:00:00:11.541 34652== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:21.959 INFO:tasks.ceph.osd.4.smithi178.stderr:==00:00:00:11.587 34485== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:21.968 INFO:tasks.ceph.osd.6.smithi178.stderr:==00:00:00:11.588 34487== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:21.970 INFO:tasks.ceph.osd.3.smithi117.stderr:==00:00:00:11.587 34651== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:21.994 INFO:tasks.ceph.osd.0.smithi117.stderr:==00:00:00:11.616 34650== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:22.458 INFO:tasks.ceph.osd.1.smithi117.stderr:==00:00:00:12.067 34652== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:22.548 INFO:tasks.ceph.osd.1.smithi117.stderr:==00:00:00:12.156 34652== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:22.609 INFO:tasks.ceph.osd.0.smithi117.stderr:==00:00:00:12.231 34650== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:22.664 INFO:tasks.ceph.osd.0.smithi117.stderr:==00:00:00:12.286 34650== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:27.293 INFO:tasks.ceph.osd.7.smithi178.stderr:==00:00:00:16.913 34488== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:27.386 INFO:tasks.ceph.osd.3.smithi117.stderr:==00:00:00:17.004 34651== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:27.444 INFO:tasks.ceph.osd.4.smithi178.stderr:==00:00:00:17.072 34485== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:27.578 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-15T12:42:27.583 INFO:tasks.ceph.osd.5.smithi178.stderr:==00:00:00:17.206 34486== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:27.676 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.686+0000 7f4b2483a640 10 monclient: get_monmap_and_config 2024-05-15T12:42:27.677 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.686+0000 7f4b2483a640 10 monclient: build_initial_monmap 2024-05-15T12:42:27.677 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.686+0000 7f4b2483a640 10 monclient: monmap: 2024-05-15T12:42:27.677 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-05-15T12:42:27.677 INFO:teuthology.misc.health.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:27.677 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-05-15T12:42:27.687750+0000 2024-05-15T12:42:27.677 INFO:teuthology.misc.health.smithi117.stderr:created 2024-05-15T12:42:27.687750+0000 2024-05-15T12:42:27.677 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:42:27.677 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-05-15T12:42:27.677 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T12:42:27.677 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T12:42:27.678 INFO:teuthology.misc.health.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T12:42:27.678 INFO:teuthology.misc.health.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T12:42:27.678 INFO:teuthology.misc.health.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T12:42:27.678 INFO:teuthology.misc.health.smithi117.stderr: 2024-05-15T12:42:27.681 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.687+0000 7f4b2483a640 1 Processor -- start 2024-05-15T12:42:27.681 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.687+0000 7f4b2483a640 1 -- start start 2024-05-15T12:42:27.682 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.687+0000 7f4b2483a640 10 monclient: init 2024-05-15T12:42:27.682 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.688+0000 7f4b2483a640 10 monclient: _reopen_session rank -1 2024-05-15T12:42:27.682 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.688+0000 7f4b2483a640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-05-15T12:42:27.682 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.688+0000 7f4b2483a640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f4b201431e0 0x7f4b200b5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:27.682 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.688+0000 7f4b2483a640 10 monclient(hunting): picked mon.noname-c con 0x7f4b201431e0 addr v2:172.21.15.178:3300/0 2024-05-15T12:42:27.682 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.688+0000 7f4b2483a640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4b200b1d10 0x7f4b200b2170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:27.682 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.688+0000 7f4b2483a640 10 monclient(hunting): picked mon.noname-b con 0x7f4b200b1d10 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:42:27.682 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.688+0000 7f4b2483a640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f4b200afd80 0x7f4b200b0210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:27.682 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.688+0000 7f4b2483a640 10 monclient(hunting): picked mon.noname-a con 0x7f4b200afd80 addr v2:172.21.15.117:3300/0 2024-05-15T12:42:27.682 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.688+0000 7f4b2483a640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:27.682 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.688+0000 7f4b2483a640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b200b2840 con 0x7f4b200afd80 2024-05-15T12:42:27.682 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.688+0000 7f4b2483a640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:27.682 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.688+0000 7f4b2483a640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b200b0750 con 0x7f4b201431e0 2024-05-15T12:42:27.683 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.688+0000 7f4b2483a640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:27.683 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.688+0000 7f4b2483a640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b200b18f0 con 0x7f4b200b1d10 2024-05-15T12:42:27.683 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.688+0000 7f4b2483a640 10 monclient(hunting): _renew_subs 2024-05-15T12:42:27.683 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.688+0000 7f4b2483a640 10 monclient(hunting): authenticate will time out at 1310.930056s 2024-05-15T12:42:27.683 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.688+0000 7f4b1e59c640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f4b201431e0 0x7f4b200b5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:27.683 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.688+0000 7f4b1dd9b640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4b200b1d10 0x7f4b200b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:27.683 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.689+0000 7f4b1e59c640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f4b201431e0 0x7f4b200b5220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:36802/0 (socket says 172.21.15.117:36802) 2024-05-15T12:42:27.683 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.689+0000 7f4b1e59c640 1 -- 172.21.15.117:0/3469348674 learned_addr learned my addr 172.21.15.117:0/3469348674 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:42:27.683 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.689+0000 7f4b1e59c640 10 monclient(hunting): get_auth_request con 0x7f4b201431e0 auth_method 0 2024-05-15T12:42:27.683 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.689+0000 7f4b1e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:27.683 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.689+0000 7f4b1e59c640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:27.683 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.689+0000 7f4b1e59c640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:27.683 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.689+0000 7f4b1dd9b640 10 monclient(hunting): get_auth_request con 0x7f4b200b1d10 auth_method 0 2024-05-15T12:42:27.683 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.689+0000 7f4b1dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:27.684 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.689+0000 7f4b1dd9b640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:27.684 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.689+0000 7f4b1dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:27.684 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.689+0000 7f4b1e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:27.684 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.689+0000 7f4b1e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:27.684 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.689+0000 7f4b1e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:27.684 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.689+0000 7f4b1dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:27.684 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.689+0000 7f4b1dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:27.684 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.689+0000 7f4b1dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:27.684 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.690+0000 7f4b1ed9d640 1 --2- 172.21.15.117:0/3469348674 >> v2:172.21.15.117:3300/0 conn(0x7f4b200afd80 0x7f4b200b0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:27.684 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.690+0000 7f4b1ed9d640 10 monclient(hunting): get_auth_request con 0x7f4b200afd80 auth_method 0 2024-05-15T12:42:27.684 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.690+0000 7f4b1ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:27.684 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.690+0000 7f4b1ed9d640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:27.684 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.690+0000 7f4b1ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:27.684 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.690+0000 7f4b1ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:27.684 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.690+0000 7f4b1ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:27.685 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.690+0000 7f4b1ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:27.685 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.691+0000 7f4b1e59c640 10 monclient(hunting): handle_auth_done global_id 4219 payload 274 2024-05-15T12:42:27.685 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.691+0000 7f4b1e59c640 1 -- 172.21.15.117:0/3469348674 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4b200b1d10 msgr2=0x7f4b200b2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:42:27.685 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.691+0000 7f4b1e59c640 1 --2- 172.21.15.117:0/3469348674 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4b200b1d10 0x7f4b200b2170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:27.685 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.691+0000 7f4b1e59c640 1 -- 172.21.15.117:0/3469348674 >> v2:172.21.15.117:3300/0 conn(0x7f4b200afd80 msgr2=0x7f4b200b0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:42:27.685 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.691+0000 7f4b1e59c640 1 --2- 172.21.15.117:0/3469348674 >> v2:172.21.15.117:3300/0 conn(0x7f4b200afd80 0x7f4b200b0210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:27.685 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.691+0000 7f4b1e59c640 10 monclient: _finish_hunting 0 2024-05-15T12:42:27.685 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.691+0000 7f4b1e59c640 1 monclient: found mon.noname-c 2024-05-15T12:42:27.685 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.691+0000 7f4b1e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:42:27.685 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.691+0000 7f4b1e59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.178:3300/0 2024-05-15T12:42:27.685 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.691+0000 7f4b1e59c640 1 -- 172.21.15.117:0/3469348674 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b200b5760 con 0x7f4b201431e0 2024-05-15T12:42:27.686 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.693+0000 7f4b1e59c640 1 --2- 172.21.15.117:0/3469348674 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4b201431e0 0x7f4b200b5220 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4b2006dee0 tx=0x7f4b1002b890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c41184d2a40b75f4 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:42:27.691 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.699+0000 7f4b1d59a640 1 -- 172.21.15.117:0/3469348674 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b100317f0 con 0x7f4b201431e0 2024-05-15T12:42:27.691 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.699+0000 7f4b1d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:27.691 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.699+0000 7f4b1d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:42:27.691 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.699+0000 7f4b1d59a640 10 monclient: dump: 2024-05-15T12:42:27.691 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-05-15T12:42:27.691 INFO:teuthology.misc.health.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:27.691 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:27.692 INFO:teuthology.misc.health.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:27.692 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:27.692 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-05-15T12:42:27.692 INFO:teuthology.misc.health.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:27.692 INFO:teuthology.misc.health.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:27.692 INFO:teuthology.misc.health.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:27.692 INFO:teuthology.misc.health.smithi117.stderr: 2024-05-15T12:42:27.692 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.699+0000 7f4b1d59a640 10 monclient: _finish_auth 0 2024-05-15T12:42:27.692 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.699+0000 7f4b1d59a640 10 monclient: _check_auth_tickets 2024-05-15T12:42:27.692 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.699+0000 7f4b1d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:42:27.692 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.699+0000 7f4b1d59a640 1 -- 172.21.15.117:0/3469348674 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b10037070 con 0x7f4b201431e0 2024-05-15T12:42:27.692 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.699+0000 7f4b2483a640 5 monclient: authenticate success, global_id 4219 2024-05-15T12:42:27.692 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.699+0000 7f4b2483a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:42:27.693 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.699+0000 7f4b1d59a640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:42:27.693 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.699+0000 7f4b1d59a640 1 -- 172.21.15.117:0/3469348674 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b10031bb0 con 0x7f4b201431e0 2024-05-15T12:42:27.693 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.699+0000 7f4b2483a640 10 monclient: get_monmap_and_config success 2024-05-15T12:42:27.693 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.699+0000 7f4b2483a640 4 set_mon_vals no callback set 2024-05-15T12:42:27.693 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.700+0000 7f4b1d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:27.693 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.700+0000 7f4b1d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:42:27.693 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.700+0000 7f4b1d59a640 10 monclient: dump: 2024-05-15T12:42:27.693 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-05-15T12:42:27.693 INFO:teuthology.misc.health.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:27.693 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:27.693 INFO:teuthology.misc.health.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:27.693 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:27.693 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-05-15T12:42:27.693 INFO:teuthology.misc.health.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:27.693 INFO:teuthology.misc.health.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:27.694 INFO:teuthology.misc.health.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:27.694 INFO:teuthology.misc.health.smithi117.stderr: 2024-05-15T12:42:27.694 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.700+0000 7f4b2483a640 10 monclient: shutdown 2024-05-15T12:42:27.694 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.700+0000 7f4b2483a640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:42:27.694 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.700+0000 7f4b2483a640 1 -- 172.21.15.117:0/3469348674 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4b201431e0 msgr2=0x7f4b200b5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:27.694 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.700+0000 7f4b2483a640 1 --2- 172.21.15.117:0/3469348674 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4b201431e0 0x7f4b200b5220 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4b2006dee0 tx=0x7f4b1002b890 comp rx=0 tx=0).stop 2024-05-15T12:42:27.694 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.700+0000 7f4b2483a640 1 -- 172.21.15.117:0/3469348674 shutdown_connections 2024-05-15T12:42:27.694 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.700+0000 7f4b2483a640 1 --2- 172.21.15.117:0/3469348674 >> v2:172.21.15.117:3300/0 conn(0x7f4b200afd80 0x7f4b200b0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:27.694 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.700+0000 7f4b2483a640 1 --2- 172.21.15.117:0/3469348674 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4b200b1d10 0x7f4b200b2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:27.694 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.700+0000 7f4b2483a640 1 --2- 172.21.15.117:0/3469348674 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4b201431e0 0x7f4b200b5220 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:27.694 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.700+0000 7f4b2483a640 1 -- 172.21.15.117:0/3469348674 >> 172.21.15.117:0/3469348674 conn(0x7f4b2006d3a0 msgr2=0x7f4b2006d7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:42:27.694 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.700+0000 7f4b2483a640 1 -- 172.21.15.117:0/3469348674 shutdown_connections 2024-05-15T12:42:27.694 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.700+0000 7f4b2483a640 1 -- 172.21.15.117:0/3469348674 wait complete. 2024-05-15T12:42:27.700 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.702+0000 7f4b2483a640 10 monclient: build_initial_monmap 2024-05-15T12:42:27.700 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.702+0000 7f4b2483a640 10 monclient: monmap: 2024-05-15T12:42:27.700 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-05-15T12:42:27.701 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:42:27.701 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-05-15T12:42:27.701 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-05-15T12:42:27.701 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:42:27.701 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-05-15T12:42:27.701 INFO:teuthology.misc.health.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T12:42:27.701 INFO:teuthology.misc.health.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T12:42:27.701 INFO:teuthology.misc.health.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T12:42:27.701 INFO:teuthology.misc.health.smithi117.stderr: 2024-05-15T12:42:27.701 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.702+0000 7f4b2483a640 1 librados: starting msgr at 2024-05-15T12:42:27.701 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.702+0000 7f4b2483a640 1 librados: starting objecter 2024-05-15T12:42:27.701 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.702+0000 7f4b2483a640 1 Processor -- start 2024-05-15T12:42:27.701 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.702+0000 7f4b2483a640 1 -- start start 2024-05-15T12:42:27.701 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.702+0000 7f4b2483a640 1 librados: setting wanted keys 2024-05-15T12:42:27.701 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.702+0000 7f4b2483a640 1 librados: calling monclient init 2024-05-15T12:42:27.702 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.702+0000 7f4b2483a640 10 monclient: init 2024-05-15T12:42:27.702 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.703+0000 7f4b2483a640 10 monclient: _reopen_session rank -1 2024-05-15T12:42:27.702 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.703+0000 7f4b2483a640 10 monclient: _add_conns ranks=[1,2,0] 2024-05-15T12:42:27.702 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.703+0000 7f4b2483a640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4b200afd80 0x7f4b200b7ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:27.702 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.703+0000 7f4b2483a640 10 monclient(hunting): picked mon.noname-b con 0x7f4b200afd80 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:42:27.702 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.703+0000 7f4b2483a640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4b200b1d10 0x7f4b201cba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:27.702 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.703+0000 7f4b2483a640 10 monclient(hunting): picked mon.noname-c con 0x7f4b200b1d10 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:42:27.702 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.704+0000 7f4b2483a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b201431e0 0x7f4b201cce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:27.702 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.704+0000 7f4b2483a640 10 monclient(hunting): picked mon.noname-a con 0x7f4b201431e0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T12:42:27.702 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.704+0000 7f4b2483a640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:27.702 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.704+0000 7f4b2483a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b200b2b50 con 0x7f4b201431e0 2024-05-15T12:42:27.702 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.704+0000 7f4b2483a640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:27.702 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.704+0000 7f4b2483a640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b201435c0 con 0x7f4b200afd80 2024-05-15T12:42:27.702 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.704+0000 7f4b2483a640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:27.703 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.704+0000 7f4b2483a640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b200b2cd0 con 0x7f4b200b1d10 2024-05-15T12:42:27.703 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.704+0000 7f4b2483a640 10 monclient(hunting): _renew_subs 2024-05-15T12:42:27.703 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.704+0000 7f4b2483a640 10 monclient(hunting): authenticate will time out at 1310.946368s 2024-05-15T12:42:27.703 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.704+0000 7f4b1dd9b640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4b200b1d10 0x7f4b201cba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:27.703 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.704+0000 7f4b1ed9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b201431e0 0x7f4b201cce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:27.703 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.705+0000 7f4b1dd9b640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4b200b1d10 0x7f4b201cba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.117:49016/0 (socket says 172.21.15.117:49016) 2024-05-15T12:42:27.703 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.705+0000 7f4b1dd9b640 1 -- 172.21.15.117:0/2819694903 learned_addr learned my addr 172.21.15.117:0/2819694903 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:42:27.703 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.705+0000 7f4b1dd9b640 10 monclient(hunting): get_auth_request con 0x7f4b200b1d10 auth_method 0 2024-05-15T12:42:27.703 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.705+0000 7f4b1dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:27.703 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.705+0000 7f4b1dd9b640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:27.703 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.705+0000 7f4b1dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:27.703 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.705+0000 7f4b1ed9d640 10 monclient(hunting): get_auth_request con 0x7f4b201431e0 auth_method 0 2024-05-15T12:42:27.703 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.705+0000 7f4b1ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:27.703 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.705+0000 7f4b1ed9d640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:27.703 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.705+0000 7f4b1ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:27.703 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.705+0000 7f4b1dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:27.703 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.705+0000 7f4b1dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:27.704 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.705+0000 7f4b1dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:27.704 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.706+0000 7f4b1e59c640 1 --2- 172.21.15.117:0/2819694903 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4b200afd80 0x7f4b200b7ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:27.704 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.706+0000 7f4b1ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:27.704 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.706+0000 7f4b1ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:27.704 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.706+0000 7f4b1ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:27.704 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.706+0000 7f4b1e59c640 10 monclient(hunting): get_auth_request con 0x7f4b200afd80 auth_method 0 2024-05-15T12:42:27.704 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.706+0000 7f4b1e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:27.704 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.706+0000 7f4b1e59c640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:27.704 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.706+0000 7f4b1e59c640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:27.704 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.706+0000 7f4b1e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:27.704 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.706+0000 7f4b1e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:27.704 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.706+0000 7f4b1e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:27.704 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.709+0000 7f4b1dd9b640 10 monclient(hunting): handle_auth_done global_id 4178 payload 824 2024-05-15T12:42:27.704 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.709+0000 7f4b1dd9b640 1 -- 172.21.15.117:0/2819694903 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4b200afd80 msgr2=0x7f4b200b7ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:27.704 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.709+0000 7f4b1dd9b640 1 --2- 172.21.15.117:0/2819694903 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4b200afd80 0x7f4b200b7ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:27.704 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.709+0000 7f4b1dd9b640 1 -- 172.21.15.117:0/2819694903 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b201431e0 msgr2=0x7f4b201cce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:27.705 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.709+0000 7f4b1dd9b640 1 --2- 172.21.15.117:0/2819694903 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b201431e0 0x7f4b201cce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:27.705 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.709+0000 7f4b1dd9b640 10 monclient: _finish_hunting 0 2024-05-15T12:42:27.705 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.709+0000 7f4b1dd9b640 1 monclient: found mon.noname-c 2024-05-15T12:42:27.705 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.709+0000 7f4b1dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:42:27.705 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.709+0000 7f4b1dd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.117:3301/0 2024-05-15T12:42:27.705 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.709+0000 7f4b1dd9b640 1 -- 172.21.15.117:0/2819694903 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b201cd3a0 con 0x7f4b200b1d10 2024-05-15T12:42:27.705 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.713+0000 7f4b1dd9b640 1 --2- 172.21.15.117:0/2819694903 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4b200b1d10 0x7f4b201cba50 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4b1400e9e0 tx=0x7f4b1400eea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:42:27.706 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.713+0000 7f4b0effd640 10 client.?.objecter ms_handle_connect 0x7f4b200b1d10 2024-05-15T12:42:27.706 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.713+0000 7f4b0effd640 10 client.?.objecter resend_mon_ops 2024-05-15T12:42:27.710 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.718+0000 7f4b0effd640 1 -- 172.21.15.117:0/2819694903 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b14017070 con 0x7f4b200b1d10 2024-05-15T12:42:27.710 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.718+0000 7f4b0effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:27.710 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.718+0000 7f4b0effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:42:27.710 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.718+0000 7f4b0effd640 10 monclient: dump: 2024-05-15T12:42:27.710 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-05-15T12:42:27.710 INFO:teuthology.misc.health.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:27.711 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:27.711 INFO:teuthology.misc.health.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:27.711 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:27.711 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-05-15T12:42:27.711 INFO:teuthology.misc.health.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:27.711 INFO:teuthology.misc.health.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:27.711 INFO:teuthology.misc.health.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:27.711 INFO:teuthology.misc.health.smithi117.stderr: 2024-05-15T12:42:27.711 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.718+0000 7f4b0effd640 10 monclient: _finish_auth 0 2024-05-15T12:42:27.711 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.718+0000 7f4b0effd640 10 monclient: _check_auth_tickets 2024-05-15T12:42:27.711 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.718+0000 7f4b0effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:42:27.711 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.718+0000 7f4b0effd640 1 -- 172.21.15.117:0/2819694903 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b1400c9d0 con 0x7f4b200b1d10 2024-05-15T12:42:27.711 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.718+0000 7f4b0effd640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:42:27.711 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.718+0000 7f4b1cd99640 4 set_mon_vals no callback set 2024-05-15T12:42:27.711 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.719+0000 7f4b2483a640 5 monclient: authenticate success, global_id 4178 2024-05-15T12:42:27.711 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.719+0000 7f4b0effd640 1 -- 172.21.15.117:0/2819694903 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b1400cdc0 con 0x7f4b200b1d10 2024-05-15T12:42:27.712 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.719+0000 7f4b0effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:27.712 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.719+0000 7f4b0effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T12:42:27.712 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.719+0000 7f4b0effd640 10 monclient: dump: 2024-05-15T12:42:27.712 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-05-15T12:42:27.712 INFO:teuthology.misc.health.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:27.712 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:27.712 INFO:teuthology.misc.health.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:27.712 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:27.712 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-05-15T12:42:27.712 INFO:teuthology.misc.health.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:27.712 INFO:teuthology.misc.health.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:27.712 INFO:teuthology.misc.health.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:27.712 INFO:teuthology.misc.health.smithi117.stderr: 2024-05-15T12:42:27.712 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.719+0000 7f4b2483a640 10 monclient: _renew_subs 2024-05-15T12:42:27.712 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.719+0000 7f4b2483a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T12:42:27.712 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.719+0000 7f4b2483a640 1 -- 172.21.15.117:0/2819694903 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b201cdd00 con 0x7f4b200b1d10 2024-05-15T12:42:27.712 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.719+0000 7f4b2483a640 10 client.4178.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:42:27.713 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.719+0000 7f4b2483a640 10 monclient: _renew_subs 2024-05-15T12:42:27.713 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.719+0000 7f4b2483a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T12:42:27.713 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.719+0000 7f4b2483a640 1 -- 172.21.15.117:0/2819694903 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b201ce200 con 0x7f4b200b1d10 2024-05-15T12:42:27.713 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.719+0000 7f4b2483a640 1 librados: init done 2024-05-15T12:42:27.713 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.721+0000 7f4b2483a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:42:27.713 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.721+0000 7f4b2483a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:42:27.714 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.721+0000 7f4b2483a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T12:42:27.714 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.721+0000 7f4b2483a640 1 -- 172.21.15.117:0/2819694903 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b201ce4c0 con 0x7f4b200b1d10 2024-05-15T12:42:27.742 INFO:tasks.ceph.osd.6.smithi178.stderr:==00:00:00:17.362 34487== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:27.743 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.754+0000 7f4b0effd640 1 -- 172.21.15.117:0/2819694903 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 4) v1 ==== 89722+0+0 (secure 0 0 0) 0x7f4b1401a070 con 0x7f4b200b1d10 2024-05-15T12:42:27.744 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.754+0000 7f4b0effd640 1 --2- 172.21.15.117:0/2819694903 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f4b0006cdd0 0x7f4b0006f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:27.744 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.754+0000 7f4b1e59c640 1 --2- 172.21.15.117:0/2819694903 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f4b0006cdd0 0x7f4b0006f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:27.744 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.754+0000 7f4b1e59c640 10 monclient: get_auth_request con 0x7f4b0006cdd0 auth_method 0 2024-05-15T12:42:27.745 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.755+0000 7f4b0effd640 10 client.4178.objecter ms_handle_connect 0x7f4b0006cdd0 2024-05-15T12:42:27.745 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.755+0000 7f4b1e59c640 1 --2- 172.21.15.117:0/2819694903 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f4b0006cdd0 0x7f4b0006f290 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4b10036000 tx=0x7f4b1003d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:42:27.749 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.759+0000 7f4b0effd640 1 -- 172.21.15.117:0/2819694903 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f4b14057e30 con 0x7f4b200b1d10 2024-05-15T12:42:27.749 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.759+0000 7f4b0effd640 10 client.4178.objecter ms_dispatch 0x7f4b20000b90 osd_map(10..10 src has 1..10) v4 2024-05-15T12:42:27.749 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.759+0000 7f4b0effd640 3 client.4178.objecter handle_osd_map got epochs [10,10] > 0 2024-05-15T12:42:27.749 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.759+0000 7f4b0effd640 3 client.4178.objecter handle_osd_map decoding full epoch 10 2024-05-15T12:42:27.749 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:27.759+0000 7f4b0effd640 20 client.4178.objecter dump_active .. 0 homeless 2024-05-15T12:42:27.763 INFO:tasks.ceph.osd.2.smithi117.stderr:==00:00:00:17.371 34653== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:28.082 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:28.092+0000 7f4b0effd640 1 -- 172.21.15.117:0/2819694903 <== mon.2 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f4b140591a0 con 0x7f4b200b1d10 2024-05-15T12:42:28.082 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:28.092+0000 7f4b0effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:42:28.082 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:28.092+0000 7f4b0effd640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:42:28.267 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:28.277+0000 7f4b2483a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:42:28.267 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:28.277+0000 7f4b2483a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:42:28.267 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:28.277+0000 7f4b2483a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T12:42:28.267 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:28.277+0000 7f4b2483a640 1 -- 172.21.15.117:0/2819694903 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4b2004f0e0 con 0x7f4b200b1d10 2024-05-15T12:42:28.284 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:28.294+0000 7f4b0effd640 1 -- 172.21.15.117:0/2819694903 <== mon.2 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f4b1405d240 con 0x7f4b200b1d10 2024-05-15T12:42:28.284 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:28.294+0000 7f4b0effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:42:28.284 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:28.294+0000 7f4b0effd640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:42:28.284 INFO:teuthology.misc.health.smithi117.stdout: 2024-05-15T12:42:28.284 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"8817f6cf-d933-4825-b972-575ef59f5ba9","created":"2024-05-15T12:42:04.264009+0000","modified":"2024-05-15T12:42:09.879357+0000","last_up_change":"0.000000","last_in_change":"2024-05-15T12:42:09.879357+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"aaffd6d9-1a00-4793-9d1f-65a30501bec4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"efd4e8ed-699c-4ccb-b989-cfadc7aaed35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"708e4e3e-7e79-480f-9dbf-7b972c2f2811","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e7b5d7d7-515c-42d9-9c41-b1ca3977865e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa647c2c-942b-4be3-afb9-6dba6b0a8cf3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5f9bc37-4bd6-4643-8f0d-e53c5f2c8782","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d5af075d-3eaa-4cb5-b6e9-6f01c201f2ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"323233c7-a3af-4fb0-ac70-de523a279b44","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-15T12:42:28.287 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:28.296+0000 7f4b2483a640 10 librados: watch_flush enter 2024-05-15T12:42:28.287 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:28.296+0000 7f4b2483a640 10 librados: watch_flush exit 2024-05-15T12:42:28.287 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:28.296+0000 7f4b2483a640 20 client.4178.objecter shutdown clearing up homeless session... 2024-05-15T12:42:28.287 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:28.296+0000 7f4b2483a640 10 client.4178.objecter successfully canceled tick 2024-05-15T12:42:28.287 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:28.296+0000 7f4b2483a640 1 -- 172.21.15.117:0/2819694903 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f4b0006cdd0 msgr2=0x7f4b0006f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:28.287 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:28.296+0000 7f4b2483a640 1 --2- 172.21.15.117:0/2819694903 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f4b0006cdd0 0x7f4b0006f290 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4b10036000 tx=0x7f4b1003d040 comp rx=0 tx=0).stop 2024-05-15T12:42:28.288 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:28.296+0000 7f4b2483a640 10 monclient: shutdown 2024-05-15T12:42:28.288 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:28.296+0000 7f4b2483a640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:42:28.288 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:28.296+0000 7f4b2483a640 1 -- 172.21.15.117:0/2819694903 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4b200b1d10 msgr2=0x7f4b201cba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:28.288 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:28.296+0000 7f4b2483a640 1 --2- 172.21.15.117:0/2819694903 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4b200b1d10 0x7f4b201cba50 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4b1400e9e0 tx=0x7f4b1400eea0 comp rx=0 tx=0).stop 2024-05-15T12:42:28.288 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:28.296+0000 7f4b2483a640 1 -- 172.21.15.117:0/2819694903 shutdown_connections 2024-05-15T12:42:28.288 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:28.296+0000 7f4b2483a640 1 --2- 172.21.15.117:0/2819694903 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b201431e0 0x7f4b201cce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:28.288 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:28.296+0000 7f4b2483a640 1 --2- 172.21.15.117:0/2819694903 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4b200b1d10 0x7f4b201cba50 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:28.288 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:28.296+0000 7f4b2483a640 1 --2- 172.21.15.117:0/2819694903 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f4b0006cdd0 0x7f4b0006f290 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:28.288 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:28.296+0000 7f4b2483a640 1 --2- 172.21.15.117:0/2819694903 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4b200afd80 0x7f4b200b7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:28.288 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:28.296+0000 7f4b2483a640 1 -- 172.21.15.117:0/2819694903 >> 172.21.15.117:0/2819694903 conn(0x7f4b2006d3a0 msgr2=0x7f4b20141dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:42:28.288 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:28.296+0000 7f4b2483a640 1 -- 172.21.15.117:0/2819694903 shutdown_connections 2024-05-15T12:42:28.288 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:28.297+0000 7f4b2483a640 1 -- 172.21.15.117:0/2819694903 wait complete. 2024-05-15T12:42:28.288 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:28.297+0000 7f4b2483a640 1 librados: shutdown 2024-05-15T12:42:28.297 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-05-15T12:42:28.352 INFO:tasks.ceph.osd.0.smithi117.stderr:==00:00:00:17.974 34650== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:28.594 INFO:tasks.ceph.osd.1.smithi117.stderr:==00:00:00:18.202 34652== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:32.440 INFO:tasks.ceph.osd.7.smithi178.stderr:==00:00:00:22.060 34488== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:32.444 INFO:tasks.ceph.osd.7.smithi178.stderr:==00:00:00:22.064 34488== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:32.702 INFO:tasks.ceph.osd.4.smithi178.stderr:==00:00:00:22.330 34485== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:32.706 INFO:tasks.ceph.osd.4.smithi178.stderr:==00:00:00:22.334 34485== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:32.814 INFO:tasks.ceph.osd.5.smithi178.stderr:==00:00:00:22.437 34486== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:32.821 INFO:tasks.ceph.osd.5.smithi178.stderr:==00:00:00:22.444 34486== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:32.902 INFO:tasks.ceph.osd.3.smithi117.stderr:==00:00:00:22.520 34651== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:32.906 INFO:tasks.ceph.osd.3.smithi117.stderr:==00:00:00:22.524 34651== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:33.028 INFO:tasks.ceph.osd.6.smithi178.stderr:==00:00:00:22.648 34487== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:33.034 INFO:tasks.ceph.osd.6.smithi178.stderr:==00:00:00:22.654 34487== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:33.295 INFO:tasks.ceph.osd.2.smithi117.stderr:==00:00:00:22.904 34653== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:33.301 INFO:tasks.ceph.osd.2.smithi117.stderr:==00:00:00:22.910 34653== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:33.444 INFO:tasks.ceph.osd.0.smithi117.stderr:==00:00:00:23.066 34650== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:33.448 INFO:tasks.ceph.osd.0.smithi117.stderr:==00:00:00:23.070 34650== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:33.670 INFO:tasks.ceph.osd.1.smithi117.stderr:==00:00:00:23.279 34652== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:33.674 INFO:tasks.ceph.osd.1.smithi117.stderr:==00:00:00:23.283 34652== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:35.104 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-15T12:42:35.198 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.208+0000 7f0a0606a640 10 monclient: get_monmap_and_config 2024-05-15T12:42:35.199 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.208+0000 7f0a0606a640 10 monclient: build_initial_monmap 2024-05-15T12:42:35.199 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.208+0000 7f0a0606a640 10 monclient: monmap: 2024-05-15T12:42:35.199 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-05-15T12:42:35.199 INFO:teuthology.misc.health.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:35.199 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-05-15T12:42:35.209709+0000 2024-05-15T12:42:35.199 INFO:teuthology.misc.health.smithi117.stderr:created 2024-05-15T12:42:35.209709+0000 2024-05-15T12:42:35.199 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:42:35.199 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-05-15T12:42:35.199 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T12:42:35.199 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T12:42:35.199 INFO:teuthology.misc.health.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T12:42:35.199 INFO:teuthology.misc.health.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T12:42:35.199 INFO:teuthology.misc.health.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T12:42:35.199 INFO:teuthology.misc.health.smithi117.stderr: 2024-05-15T12:42:35.203 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.209+0000 7f0a0606a640 1 Processor -- start 2024-05-15T12:42:35.203 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.209+0000 7f0a0606a640 1 -- start start 2024-05-15T12:42:35.204 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.209+0000 7f0a0606a640 10 monclient: init 2024-05-15T12:42:35.204 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.210+0000 7f0a0606a640 10 monclient: _reopen_session rank -1 2024-05-15T12:42:35.204 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.210+0000 7f0a0606a640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-05-15T12:42:35.204 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.210+0000 7f0a0606a640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0a000b54b0 0x7f0a000a9980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:35.204 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.210+0000 7f0a0606a640 10 monclient(hunting): picked mon.noname-b con 0x7f0a000b54b0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:42:35.204 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.210+0000 7f0a0606a640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f0a001426a0 0x7f0a00142ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:35.204 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.210+0000 7f0a0606a640 10 monclient(hunting): picked mon.noname-c con 0x7f0a001426a0 addr v2:172.21.15.178:3300/0 2024-05-15T12:42:35.204 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.210+0000 7f0a0606a640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f0a0013f850 0x7f0a0013fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:35.204 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.210+0000 7f0a0606a640 10 monclient(hunting): picked mon.noname-a con 0x7f0a0013f850 addr v2:172.21.15.117:3300/0 2024-05-15T12:42:35.204 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.210+0000 7f0a0606a640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:35.204 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.210+0000 7f0a0606a640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a000b5890 con 0x7f0a0013f850 2024-05-15T12:42:35.204 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.210+0000 7f0a0606a640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:35.204 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.210+0000 7f0a0606a640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a000ab770 con 0x7f0a001426a0 2024-05-15T12:42:35.204 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.210+0000 7f0a0606a640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:35.205 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.210+0000 7f0a0606a640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a000aa050 con 0x7f0a000b54b0 2024-05-15T12:42:35.205 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.210+0000 7f0a0606a640 10 monclient(hunting): _renew_subs 2024-05-15T12:42:35.205 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.210+0000 7f0a0606a640 10 monclient(hunting): authenticate will time out at 1318.452168s 2024-05-15T12:42:35.205 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.210+0000 7f09ff7fe640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0a000b54b0 0x7f0a000a9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:35.205 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.211+0000 7f09feffd640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f0a001426a0 0x7f0a00142ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:35.205 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.211+0000 7f09fffff640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f0a0013f850 0x7f0a0013fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:35.205 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.211+0000 7f09feffd640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f0a001426a0 0x7f0a00142ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:35610/0 (socket says 172.21.15.117:35610) 2024-05-15T12:42:35.205 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.211+0000 7f09feffd640 1 -- 172.21.15.117:0/3474526198 learned_addr learned my addr 172.21.15.117:0/3474526198 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:42:35.205 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.211+0000 7f09feffd640 10 monclient(hunting): get_auth_request con 0x7f0a001426a0 auth_method 0 2024-05-15T12:42:35.205 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.211+0000 7f09feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:35.205 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.211+0000 7f09feffd640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:35.205 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.211+0000 7f09feffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:35.205 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.211+0000 7f09ff7fe640 10 monclient(hunting): get_auth_request con 0x7f0a000b54b0 auth_method 0 2024-05-15T12:42:35.205 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.211+0000 7f09ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:35.205 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.211+0000 7f09ff7fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:35.206 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.211+0000 7f09ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:35.206 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.211+0000 7f09fffff640 10 monclient(hunting): get_auth_request con 0x7f0a0013f850 auth_method 0 2024-05-15T12:42:35.206 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.211+0000 7f09fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:35.206 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.211+0000 7f09fffff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:35.206 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.211+0000 7f09fffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:35.206 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.212+0000 7f09feffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:35.206 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.212+0000 7f09feffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:35.206 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.212+0000 7f09feffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:35.206 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.212+0000 7f09ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:35.206 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.212+0000 7f09ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:35.206 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.212+0000 7f09ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:35.206 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.212+0000 7f09fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:35.206 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.212+0000 7f09fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:35.206 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.212+0000 7f09fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:35.208 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.214+0000 7f09feffd640 10 monclient(hunting): handle_auth_done global_id 4225 payload 274 2024-05-15T12:42:35.208 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.214+0000 7f09feffd640 1 -- 172.21.15.117:0/3474526198 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0a000b54b0 msgr2=0x7f0a000a9980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:42:35.208 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.214+0000 7f09feffd640 1 --2- 172.21.15.117:0/3474526198 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0a000b54b0 0x7f0a000a9980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:35.208 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.214+0000 7f09feffd640 1 -- 172.21.15.117:0/3474526198 >> v2:172.21.15.117:3300/0 conn(0x7f0a0013f850 msgr2=0x7f0a0013fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:42:35.208 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.214+0000 7f09feffd640 1 --2- 172.21.15.117:0/3474526198 >> v2:172.21.15.117:3300/0 conn(0x7f0a0013f850 0x7f0a0013fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:35.208 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.214+0000 7f09feffd640 10 monclient: _finish_hunting 0 2024-05-15T12:42:35.208 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.214+0000 7f09feffd640 1 monclient: found mon.noname-c 2024-05-15T12:42:35.208 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.214+0000 7f09feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:42:35.208 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.214+0000 7f09feffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.178:3300/0 2024-05-15T12:42:35.208 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.214+0000 7f09feffd640 1 -- 172.21.15.117:0/3474526198 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a00140220 con 0x7f0a001426a0 2024-05-15T12:42:35.208 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.217+0000 7f09feffd640 1 --2- 172.21.15.117:0/3474526198 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a001426a0 0x7f0a00142ae0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0a000b91e0 tx=0x7f09f402b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4862c14375ab653 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:42:35.212 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.222+0000 7f09fe7fc640 1 -- 172.21.15.117:0/3474526198 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f402e070 con 0x7f0a001426a0 2024-05-15T12:42:35.212 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.222+0000 7f09fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:35.212 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.222+0000 7f09fe7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:42:35.212 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.222+0000 7f09fe7fc640 10 monclient: dump: 2024-05-15T12:42:35.212 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-05-15T12:42:35.212 INFO:teuthology.misc.health.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:35.212 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:35.213 INFO:teuthology.misc.health.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:35.213 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:35.213 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-05-15T12:42:35.213 INFO:teuthology.misc.health.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:35.213 INFO:teuthology.misc.health.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:35.213 INFO:teuthology.misc.health.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:35.213 INFO:teuthology.misc.health.smithi117.stderr: 2024-05-15T12:42:35.213 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.222+0000 7f09fe7fc640 10 monclient: _finish_auth 0 2024-05-15T12:42:35.213 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.222+0000 7f09fe7fc640 10 monclient: _check_auth_tickets 2024-05-15T12:42:35.213 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.222+0000 7f09fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:42:35.213 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.222+0000 7f09fe7fc640 1 -- 172.21.15.117:0/3474526198 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09f4037070 con 0x7f0a001426a0 2024-05-15T12:42:35.213 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.222+0000 7f0a0606a640 5 monclient: authenticate success, global_id 4225 2024-05-15T12:42:35.213 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.222+0000 7f0a0606a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:42:35.213 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.222+0000 7f09fe7fc640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:42:35.213 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.222+0000 7f0a0606a640 10 monclient: get_monmap_and_config success 2024-05-15T12:42:35.214 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.222+0000 7f09fe7fc640 1 -- 172.21.15.117:0/3474526198 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f4031a00 con 0x7f0a001426a0 2024-05-15T12:42:35.214 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.222+0000 7f09fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:35.214 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.222+0000 7f0a0606a640 4 set_mon_vals no callback set 2024-05-15T12:42:35.214 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.222+0000 7f09fe7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:42:35.214 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.222+0000 7f09fe7fc640 10 monclient: dump: 2024-05-15T12:42:35.214 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-05-15T12:42:35.214 INFO:teuthology.misc.health.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:35.214 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:35.214 INFO:teuthology.misc.health.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:35.214 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:35.214 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-05-15T12:42:35.214 INFO:teuthology.misc.health.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:35.214 INFO:teuthology.misc.health.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:35.214 INFO:teuthology.misc.health.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:35.214 INFO:teuthology.misc.health.smithi117.stderr: 2024-05-15T12:42:35.220 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.222+0000 7f0a0606a640 10 monclient: shutdown 2024-05-15T12:42:35.220 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.222+0000 7f0a0606a640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:42:35.220 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.222+0000 7f0a0606a640 1 -- 172.21.15.117:0/3474526198 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a001426a0 msgr2=0x7f0a00142ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:35.220 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.222+0000 7f0a0606a640 1 --2- 172.21.15.117:0/3474526198 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a001426a0 0x7f0a00142ae0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0a000b91e0 tx=0x7f09f402b880 comp rx=0 tx=0).stop 2024-05-15T12:42:35.220 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.223+0000 7f0a0606a640 1 -- 172.21.15.117:0/3474526198 shutdown_connections 2024-05-15T12:42:35.220 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.223+0000 7f0a0606a640 1 --2- 172.21.15.117:0/3474526198 >> v2:172.21.15.117:3300/0 conn(0x7f0a0013f850 0x7f0a0013fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:35.220 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.223+0000 7f0a0606a640 1 --2- 172.21.15.117:0/3474526198 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a001426a0 0x7f0a00142ae0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:35.220 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.223+0000 7f0a0606a640 1 --2- 172.21.15.117:0/3474526198 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0a000b54b0 0x7f0a000a9980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:35.220 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.223+0000 7f0a0606a640 1 -- 172.21.15.117:0/3474526198 >> 172.21.15.117:0/3474526198 conn(0x7f0a0006d370 msgr2=0x7f0a0006d780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:42:35.221 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.223+0000 7f0a0606a640 1 -- 172.21.15.117:0/3474526198 shutdown_connections 2024-05-15T12:42:35.221 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.223+0000 7f0a0606a640 1 -- 172.21.15.117:0/3474526198 wait complete. 2024-05-15T12:42:35.221 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.223+0000 7f0a0606a640 10 monclient: build_initial_monmap 2024-05-15T12:42:35.221 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.223+0000 7f0a0606a640 10 monclient: monmap: 2024-05-15T12:42:35.221 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-05-15T12:42:35.221 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:42:35.221 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-05-15T12:42:35.221 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-05-15T12:42:35.221 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:42:35.221 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-05-15T12:42:35.221 INFO:teuthology.misc.health.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T12:42:35.221 INFO:teuthology.misc.health.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T12:42:35.221 INFO:teuthology.misc.health.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T12:42:35.222 INFO:teuthology.misc.health.smithi117.stderr: 2024-05-15T12:42:35.222 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.224+0000 7f0a0606a640 1 librados: starting msgr at 2024-05-15T12:42:35.222 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.224+0000 7f0a0606a640 1 librados: starting objecter 2024-05-15T12:42:35.222 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.224+0000 7f0a0606a640 1 Processor -- start 2024-05-15T12:42:35.222 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.224+0000 7f0a0606a640 1 -- start start 2024-05-15T12:42:35.222 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.224+0000 7f0a0606a640 1 librados: setting wanted keys 2024-05-15T12:42:35.222 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.224+0000 7f0a0606a640 1 librados: calling monclient init 2024-05-15T12:42:35.222 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.224+0000 7f0a0606a640 10 monclient: init 2024-05-15T12:42:35.222 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.225+0000 7f0a0606a640 10 monclient: _reopen_session rank -1 2024-05-15T12:42:35.222 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.225+0000 7f0a0606a640 10 monclient: _add_conns ranks=[2,1,0] 2024-05-15T12:42:35.222 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.225+0000 7f0a0606a640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0a000b54b0 0x7f0a00145d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:35.222 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.225+0000 7f0a0606a640 10 monclient(hunting): picked mon.noname-c con 0x7f0a000b54b0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:42:35.222 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.225+0000 7f0a0606a640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a0013f850 0x7f0a000b7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:35.222 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.225+0000 7f0a0606a640 10 monclient(hunting): picked mon.noname-b con 0x7f0a0013f850 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:42:35.223 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.225+0000 7f0a0606a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a001426a0 0x7f0a001dac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:35.223 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.225+0000 7f0a0606a640 10 monclient(hunting): picked mon.noname-a con 0x7f0a001426a0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T12:42:35.223 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.225+0000 7f0a0606a640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:35.223 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.225+0000 7f0a0606a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a000b48a0 con 0x7f0a001426a0 2024-05-15T12:42:35.223 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.225+0000 7f0a0606a640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:35.223 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.225+0000 7f0a0606a640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a000aa570 con 0x7f0a0013f850 2024-05-15T12:42:35.223 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.225+0000 7f0a0606a640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:35.223 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.225+0000 7f0a0606a640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a000b4a20 con 0x7f0a000b54b0 2024-05-15T12:42:35.223 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.225+0000 7f0a0606a640 10 monclient(hunting): _renew_subs 2024-05-15T12:42:35.223 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.225+0000 7f0a0606a640 10 monclient(hunting): authenticate will time out at 1318.467298s 2024-05-15T12:42:35.223 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.225+0000 7f09ff7fe640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0a000b54b0 0x7f0a00145d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:35.223 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.226+0000 7f09fffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a001426a0 0x7f0a001dac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:35.223 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.226+0000 7f09feffd640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a0013f850 0x7f0a000b7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:35.224 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.226+0000 7f09ff7fe640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0a000b54b0 0x7f0a00145d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.117:39092/0 (socket says 172.21.15.117:39092) 2024-05-15T12:42:35.224 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.226+0000 7f09ff7fe640 1 -- 172.21.15.117:0/4253722693 learned_addr learned my addr 172.21.15.117:0/4253722693 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:42:35.224 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.226+0000 7f09ff7fe640 10 monclient(hunting): get_auth_request con 0x7f0a000b54b0 auth_method 0 2024-05-15T12:42:35.224 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.226+0000 7f09ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:35.224 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.226+0000 7f09ff7fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:35.224 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.226+0000 7f09ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:35.224 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.226+0000 7f09feffd640 10 monclient(hunting): get_auth_request con 0x7f0a0013f850 auth_method 0 2024-05-15T12:42:35.224 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.226+0000 7f09feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:35.224 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.226+0000 7f09feffd640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:35.224 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.226+0000 7f09feffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:35.224 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.226+0000 7f09fffff640 10 monclient(hunting): get_auth_request con 0x7f0a001426a0 auth_method 0 2024-05-15T12:42:35.224 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.226+0000 7f09fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:35.224 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.226+0000 7f09fffff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:35.224 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.226+0000 7f09fffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:35.224 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.226+0000 7f09ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:35.225 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.226+0000 7f09ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:35.225 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.226+0000 7f09ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:35.225 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.227+0000 7f09feffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:35.225 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.227+0000 7f09feffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:35.225 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.227+0000 7f09feffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:35.225 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.227+0000 7f09fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:35.225 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.227+0000 7f09fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:35.225 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.227+0000 7f09fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:35.225 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.229+0000 7f09ff7fe640 10 monclient(hunting): handle_auth_done global_id 4184 payload 824 2024-05-15T12:42:35.225 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.229+0000 7f09ff7fe640 1 -- 172.21.15.117:0/4253722693 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a0013f850 msgr2=0x7f0a000b7410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:35.225 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.229+0000 7f09ff7fe640 1 --2- 172.21.15.117:0/4253722693 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a0013f850 0x7f0a000b7410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:35.225 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.229+0000 7f09ff7fe640 1 -- 172.21.15.117:0/4253722693 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a001426a0 msgr2=0x7f0a001dac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:35.225 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.229+0000 7f09ff7fe640 1 --2- 172.21.15.117:0/4253722693 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a001426a0 0x7f0a001dac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:35.226 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.229+0000 7f09ff7fe640 10 monclient: _finish_hunting 0 2024-05-15T12:42:35.226 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.229+0000 7f09ff7fe640 1 monclient: found mon.noname-c 2024-05-15T12:42:35.226 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.229+0000 7f09ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:42:35.226 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.229+0000 7f09ff7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.117:3301/0 2024-05-15T12:42:35.226 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.229+0000 7f09ff7fe640 1 -- 172.21.15.117:0/4253722693 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a000b8050 con 0x7f0a000b54b0 2024-05-15T12:42:35.227 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.232+0000 7f09ff7fe640 1 --2- 172.21.15.117:0/4253722693 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0a000b54b0 0x7f0a00145d30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f09f000e9e0 tx=0x7f09f000eea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:42:35.227 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.232+0000 7f09e7fff640 10 client.?.objecter ms_handle_connect 0x7f0a000b54b0 2024-05-15T12:42:35.227 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.232+0000 7f09e7fff640 10 client.?.objecter resend_mon_ops 2024-05-15T12:42:35.231 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.237+0000 7f09e7fff640 1 -- 172.21.15.117:0/4253722693 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f0017070 con 0x7f0a000b54b0 2024-05-15T12:42:35.231 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.237+0000 7f09e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:35.231 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.237+0000 7f09e7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:42:35.231 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.237+0000 7f09e7fff640 10 monclient: dump: 2024-05-15T12:42:35.231 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-05-15T12:42:35.232 INFO:teuthology.misc.health.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:35.232 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:35.232 INFO:teuthology.misc.health.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:35.232 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:35.232 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-05-15T12:42:35.232 INFO:teuthology.misc.health.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:35.232 INFO:teuthology.misc.health.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:35.232 INFO:teuthology.misc.health.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:35.232 INFO:teuthology.misc.health.smithi117.stderr: 2024-05-15T12:42:35.232 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.237+0000 7f09e7fff640 10 monclient: _finish_auth 0 2024-05-15T12:42:35.232 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.237+0000 7f09e7fff640 10 monclient: _check_auth_tickets 2024-05-15T12:42:35.232 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.237+0000 7f09e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:42:35.232 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.237+0000 7f09e7fff640 1 -- 172.21.15.117:0/4253722693 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09f000c9d0 con 0x7f0a000b54b0 2024-05-15T12:42:35.232 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.237+0000 7f09e7fff640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:42:35.232 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.237+0000 7f09e7fff640 1 -- 172.21.15.117:0/4253722693 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f000cdc0 con 0x7f0a000b54b0 2024-05-15T12:42:35.233 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.237+0000 7f09e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:35.233 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.237+0000 7f09e7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T12:42:35.233 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.237+0000 7f09e7fff640 10 monclient: dump: 2024-05-15T12:42:35.233 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-05-15T12:42:35.233 INFO:teuthology.misc.health.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:35.233 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:35.233 INFO:teuthology.misc.health.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:35.233 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:35.233 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-05-15T12:42:35.233 INFO:teuthology.misc.health.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:35.233 INFO:teuthology.misc.health.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:35.233 INFO:teuthology.misc.health.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:35.233 INFO:teuthology.misc.health.smithi117.stderr: 2024-05-15T12:42:35.233 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.237+0000 7f09fdffb640 4 set_mon_vals no callback set 2024-05-15T12:42:35.234 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.238+0000 7f0a0606a640 5 monclient: authenticate success, global_id 4184 2024-05-15T12:42:35.234 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.238+0000 7f0a0606a640 10 monclient: _renew_subs 2024-05-15T12:42:35.234 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.238+0000 7f0a0606a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T12:42:35.234 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.238+0000 7f0a0606a640 1 -- 172.21.15.117:0/4253722693 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a000b8e50 con 0x7f0a000b54b0 2024-05-15T12:42:35.234 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.238+0000 7f0a0606a640 10 client.4184.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:42:35.234 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.238+0000 7f0a0606a640 10 monclient: _renew_subs 2024-05-15T12:42:35.234 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.239+0000 7f0a0606a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T12:42:35.234 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.239+0000 7f0a0606a640 1 -- 172.21.15.117:0/4253722693 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a000b8a40 con 0x7f0a000b54b0 2024-05-15T12:42:35.234 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.239+0000 7f0a0606a640 1 librados: init done 2024-05-15T12:42:35.234 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.240+0000 7f0a0606a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:42:35.234 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.240+0000 7f0a0606a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:42:35.234 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.240+0000 7f0a0606a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T12:42:35.234 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.240+0000 7f0a0606a640 1 -- 172.21.15.117:0/4253722693 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a00068a70 con 0x7f0a000b54b0 2024-05-15T12:42:35.271 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.280+0000 7f09e7fff640 1 -- 172.21.15.117:0/4253722693 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 4) v1 ==== 89722+0+0 (secure 0 0 0) 0x7f09f001a070 con 0x7f0a000b54b0 2024-05-15T12:42:35.271 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.281+0000 7f09e7fff640 1 --2- 172.21.15.117:0/4253722693 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f09ec06ce20 0x7f09ec06f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:35.271 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.281+0000 7f09feffd640 1 --2- 172.21.15.117:0/4253722693 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f09ec06ce20 0x7f09ec06f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:35.271 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.281+0000 7f09feffd640 10 monclient: get_auth_request con 0x7f09ec06ce20 auth_method 0 2024-05-15T12:42:35.272 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.282+0000 7f09e7fff640 10 client.4184.objecter ms_handle_connect 0x7f09ec06ce20 2024-05-15T12:42:35.272 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.282+0000 7f09feffd640 1 --2- 172.21.15.117:0/4253722693 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f09ec06ce20 0x7f09ec06f2e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f09f4036000 tx=0x7f09f403d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:42:35.490 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.499+0000 7f09e7fff640 1 -- 172.21.15.117:0/4253722693 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f09f0057dc0 con 0x7f0a000b54b0 2024-05-15T12:42:35.490 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.499+0000 7f09e7fff640 10 client.4184.objecter ms_dispatch 0x7f0a00000b90 osd_map(10..10 src has 1..10) v4 2024-05-15T12:42:35.490 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.499+0000 7f09e7fff640 3 client.4184.objecter handle_osd_map got epochs [10,10] > 0 2024-05-15T12:42:35.490 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.499+0000 7f09e7fff640 3 client.4184.objecter handle_osd_map decoding full epoch 10 2024-05-15T12:42:35.490 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.499+0000 7f09e7fff640 20 client.4184.objecter dump_active .. 0 homeless 2024-05-15T12:42:35.503 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.513+0000 7f09e7fff640 1 -- 172.21.15.117:0/4253722693 <== mon.2 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f09f0059130 con 0x7f0a000b54b0 2024-05-15T12:42:35.503 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.513+0000 7f09e7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:42:35.503 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.513+0000 7f09e7fff640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:42:35.689 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.698+0000 7f0a0606a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:42:35.689 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.698+0000 7f0a0606a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:42:35.689 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.698+0000 7f0a0606a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T12:42:35.689 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.698+0000 7f0a0606a640 1 -- 172.21.15.117:0/4253722693 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0a000675f0 con 0x7f0a000b54b0 2024-05-15T12:42:35.712 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.722+0000 7f09e7fff640 1 -- 172.21.15.117:0/4253722693 <== mon.2 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f09f005cf90 con 0x7f0a000b54b0 2024-05-15T12:42:35.712 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.722+0000 7f09e7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:42:35.712 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.722+0000 7f09e7fff640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:42:35.713 INFO:teuthology.misc.health.smithi117.stdout: 2024-05-15T12:42:35.713 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"8817f6cf-d933-4825-b972-575ef59f5ba9","created":"2024-05-15T12:42:04.264009+0000","modified":"2024-05-15T12:42:09.879357+0000","last_up_change":"0.000000","last_in_change":"2024-05-15T12:42:09.879357+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"aaffd6d9-1a00-4793-9d1f-65a30501bec4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"efd4e8ed-699c-4ccb-b989-cfadc7aaed35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"708e4e3e-7e79-480f-9dbf-7b972c2f2811","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e7b5d7d7-515c-42d9-9c41-b1ca3977865e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa647c2c-942b-4be3-afb9-6dba6b0a8cf3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5f9bc37-4bd6-4643-8f0d-e53c5f2c8782","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d5af075d-3eaa-4cb5-b6e9-6f01c201f2ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"323233c7-a3af-4fb0-ac70-de523a279b44","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-15T12:42:35.716 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.724+0000 7f0a0606a640 10 librados: watch_flush enter 2024-05-15T12:42:35.716 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.724+0000 7f0a0606a640 10 librados: watch_flush exit 2024-05-15T12:42:35.716 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.724+0000 7f0a0606a640 20 client.4184.objecter shutdown clearing up homeless session... 2024-05-15T12:42:35.716 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.724+0000 7f0a0606a640 10 client.4184.objecter successfully canceled tick 2024-05-15T12:42:35.716 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.724+0000 7f0a0606a640 1 -- 172.21.15.117:0/4253722693 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f09ec06ce20 msgr2=0x7f09ec06f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:35.716 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.724+0000 7f0a0606a640 1 --2- 172.21.15.117:0/4253722693 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f09ec06ce20 0x7f09ec06f2e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f09f4036000 tx=0x7f09f403d040 comp rx=0 tx=0).stop 2024-05-15T12:42:35.716 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.724+0000 7f0a0606a640 10 monclient: shutdown 2024-05-15T12:42:35.716 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.725+0000 7f0a0606a640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:42:35.716 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.725+0000 7f0a0606a640 1 -- 172.21.15.117:0/4253722693 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0a000b54b0 msgr2=0x7f0a00145d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:35.716 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.725+0000 7f0a0606a640 1 --2- 172.21.15.117:0/4253722693 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0a000b54b0 0x7f0a00145d30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f09f000e9e0 tx=0x7f09f000eea0 comp rx=0 tx=0).stop 2024-05-15T12:42:35.716 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.725+0000 7f0a0606a640 1 -- 172.21.15.117:0/4253722693 shutdown_connections 2024-05-15T12:42:35.717 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.725+0000 7f0a0606a640 1 --2- 172.21.15.117:0/4253722693 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a001426a0 0x7f0a001dac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:35.717 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.725+0000 7f0a0606a640 1 --2- 172.21.15.117:0/4253722693 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f09ec06ce20 0x7f09ec06f2e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:35.717 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.725+0000 7f0a0606a640 1 --2- 172.21.15.117:0/4253722693 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a0013f850 0x7f0a000b7410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:35.717 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.725+0000 7f0a0606a640 1 --2- 172.21.15.117:0/4253722693 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0a000b54b0 0x7f0a00145d30 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:35.717 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.725+0000 7f0a0606a640 1 -- 172.21.15.117:0/4253722693 >> 172.21.15.117:0/4253722693 conn(0x7f0a0006d370 msgr2=0x7f0a000a94d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:42:35.717 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.725+0000 7f0a0606a640 1 -- 172.21.15.117:0/4253722693 shutdown_connections 2024-05-15T12:42:35.717 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.725+0000 7f0a0606a640 1 -- 172.21.15.117:0/4253722693 wait complete. 2024-05-15T12:42:35.717 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:35.725+0000 7f0a0606a640 1 librados: shutdown 2024-05-15T12:42:35.727 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-05-15T12:42:36.989 INFO:tasks.ceph.osd.3.smithi117.stderr:==00:00:00:26.607 34651== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:37.039 INFO:tasks.ceph.osd.5.smithi178.stderr:==00:00:00:26.661 34486== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:37.133 INFO:tasks.ceph.osd.7.smithi178.stderr:==00:00:00:26.752 34488== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:37.390 INFO:tasks.ceph.osd.4.smithi178.stderr:==00:00:00:27.018 34485== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:37.650 INFO:tasks.ceph.osd.2.smithi117.stderr:==00:00:00:27.258 34653== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:37.725 INFO:tasks.ceph.osd.6.smithi178.stderr:==00:00:00:27.345 34487== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:37.884 INFO:tasks.ceph.osd.1.smithi117.stderr:==00:00:00:27.493 34652== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:37.884 INFO:tasks.ceph.osd.0.smithi117.stderr:==00:00:00:27.505 34650== Warning: unimplemented fcntl command: 1038 2024-05-15T12:42:42.534 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-15T12:42:42.627 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.636+0000 7f8ab0301640 10 monclient: get_monmap_and_config 2024-05-15T12:42:42.627 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.636+0000 7f8ab0301640 10 monclient: build_initial_monmap 2024-05-15T12:42:42.627 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.636+0000 7f8ab0301640 10 monclient: monmap: 2024-05-15T12:42:42.627 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-05-15T12:42:42.627 INFO:teuthology.misc.health.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:42.627 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-05-15T12:42:42.637877+0000 2024-05-15T12:42:42.627 INFO:teuthology.misc.health.smithi117.stderr:created 2024-05-15T12:42:42.637877+0000 2024-05-15T12:42:42.627 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:42:42.627 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-05-15T12:42:42.627 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T12:42:42.628 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T12:42:42.628 INFO:teuthology.misc.health.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T12:42:42.628 INFO:teuthology.misc.health.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T12:42:42.628 INFO:teuthology.misc.health.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T12:42:42.628 INFO:teuthology.misc.health.smithi117.stderr: 2024-05-15T12:42:42.630 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.637+0000 7f8ab0301640 1 Processor -- start 2024-05-15T12:42:42.630 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.637+0000 7f8ab0301640 1 -- start start 2024-05-15T12:42:42.630 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.637+0000 7f8ab0301640 10 monclient: init 2024-05-15T12:42:42.630 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.638+0000 7f8ab0301640 10 monclient: _reopen_session rank -1 2024-05-15T12:42:42.630 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.638+0000 7f8ab0301640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-05-15T12:42:42.630 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.638+0000 7f8ab0301640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8aa81431e0 addr v1:172.21.15.178:6789/0 2024-05-15T12:42:42.631 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.638+0000 7f8ab0301640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f8aa80b1d10 0x7f8aa80b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:42.631 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.638+0000 7f8ab0301640 10 monclient(hunting): picked mon.noname-a con 0x7f8aa80b1d10 addr v2:172.21.15.117:3300/0 2024-05-15T12:42:42.631 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.638+0000 7f8ab0301640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f8aa80afd80 0x7f8aa80b01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:42.631 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.638+0000 7f8ab0301640 10 monclient(hunting): picked mon.noname-c con 0x7f8aa80afd80 addr v2:172.21.15.178:3300/0 2024-05-15T12:42:42.631 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.638+0000 7f8ab0301640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8aa80b18f0 con 0x7f8aa81431e0 2024-05-15T12:42:42.631 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.638+0000 7f8ab0301640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:42.631 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.638+0000 7f8ab0301640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8aa80b2840 con 0x7f8aa80b1d10 2024-05-15T12:42:42.631 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.638+0000 7f8ab0301640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:42.631 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.638+0000 7f8ab0301640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8aa80b0710 con 0x7f8aa80afd80 2024-05-15T12:42:42.631 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.638+0000 7f8ab0301640 10 monclient(hunting): _renew_subs 2024-05-15T12:42:42.631 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.638+0000 7f8ab0301640 10 monclient(hunting): authenticate will time out at 1325.880428s 2024-05-15T12:42:42.631 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.638+0000 7f8aad89c640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f8aa80b1d10 0x7f8aa80b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:42.631 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.639+0000 7f8aae09d640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f8aa81431e0 0x7f8aa80b5220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.117:52912/0 (socket says 172.21.15.117:52912) 2024-05-15T12:42:42.631 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.639+0000 7f8aae09d640 1 -- 172.21.15.117:0/3075535206 learned_addr learned my addr 172.21.15.117:0/3075535206 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-05-15T12:42:42.631 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.639+0000 7f8aad89c640 10 monclient(hunting): get_auth_request con 0x7f8aa80b1d10 auth_method 0 2024-05-15T12:42:42.632 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.639+0000 7f8aad89c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:42.632 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.639+0000 7f8aad89c640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:42.632 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.639+0000 7f8aad89c640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:42.632 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.640+0000 7f8aad89c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:42.632 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.640+0000 7f8aad89c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:42.632 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.640+0000 7f8aad89c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:42.632 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.641+0000 7f8aad89c640 10 monclient(hunting): handle_auth_done global_id 4203 payload 274 2024-05-15T12:42:42.632 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.641+0000 7f8aad89c640 1 -- 172.21.15.117:0/3075535206 >> v2:172.21.15.178:3300/0 conn(0x7f8aa80afd80 msgr2=0x7f8aa80b01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:42:42.632 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.641+0000 7f8aad89c640 1 --2- 172.21.15.117:0/3075535206 >> v2:172.21.15.178:3300/0 conn(0x7f8aa80afd80 0x7f8aa80b01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:42.633 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.641+0000 7f8aad89c640 1 -- 172.21.15.117:0/3075535206 >> v1:172.21.15.178:6789/0 conn(0x7f8aa81431e0 legacy=0x7f8aa80b5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:42.633 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.641+0000 7f8aad89c640 10 monclient: _finish_hunting 0 2024-05-15T12:42:42.633 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.641+0000 7f8aad89c640 1 monclient: found mon.noname-a 2024-05-15T12:42:42.633 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.641+0000 7f8aad89c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:42:42.633 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.641+0000 7f8aad89c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.117:3300/0 2024-05-15T12:42:42.633 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.641+0000 7f8aad89c640 1 -- 172.21.15.117:0/3075535206 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aa81427a0 con 0x7f8aa80b1d10 2024-05-15T12:42:42.633 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.643+0000 7f8aad89c640 1 --2- 172.21.15.117:0/3075535206 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8aa80b1d10 0x7f8aa80b21a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f8aa806cf10 tx=0x7f8a9c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e04f3f97648fc21 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:42:42.638 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.647+0000 7f8aad09b640 1 -- 172.21.15.117:0/3075535206 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a9c0315a0 con 0x7f8aa80b1d10 2024-05-15T12:42:42.639 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.647+0000 7f8aad09b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:42.639 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.647+0000 7f8aad09b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T12:42:42.639 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.647+0000 7f8aad09b640 10 monclient: dump: 2024-05-15T12:42:42.639 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-05-15T12:42:42.639 INFO:teuthology.misc.health.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:42.639 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:42.639 INFO:teuthology.misc.health.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:42.639 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:42.639 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-05-15T12:42:42.639 INFO:teuthology.misc.health.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:42.639 INFO:teuthology.misc.health.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:42.639 INFO:teuthology.misc.health.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:42.639 INFO:teuthology.misc.health.smithi117.stderr: 2024-05-15T12:42:42.639 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.647+0000 7f8aad09b640 10 monclient: _finish_auth 0 2024-05-15T12:42:42.640 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.647+0000 7f8aad09b640 10 monclient: _check_auth_tickets 2024-05-15T12:42:42.640 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.647+0000 7f8aad09b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:42:42.640 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.647+0000 7f8ab0301640 5 monclient: authenticate success, global_id 4203 2024-05-15T12:42:42.640 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.647+0000 7f8aad09b640 1 -- 172.21.15.117:0/3075535206 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a9c037070 con 0x7f8aa80b1d10 2024-05-15T12:42:42.640 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.647+0000 7f8ab0301640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:42:42.640 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.647+0000 7f8aad09b640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:42:42.640 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.647+0000 7f8ab0301640 10 monclient: get_monmap_and_config success 2024-05-15T12:42:42.640 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.647+0000 7f8aad09b640 1 -- 172.21.15.117:0/3075535206 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a9c031960 con 0x7f8aa80b1d10 2024-05-15T12:42:42.640 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.647+0000 7f8aad09b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:42.640 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.647+0000 7f8ab0301640 4 set_mon_vals no callback set 2024-05-15T12:42:42.640 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.647+0000 7f8aad09b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T12:42:42.640 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.647+0000 7f8aad09b640 10 monclient: dump: 2024-05-15T12:42:42.640 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-05-15T12:42:42.640 INFO:teuthology.misc.health.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:42.640 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:42.641 INFO:teuthology.misc.health.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:42.641 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:42.641 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-05-15T12:42:42.641 INFO:teuthology.misc.health.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:42.641 INFO:teuthology.misc.health.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:42.641 INFO:teuthology.misc.health.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:42.641 INFO:teuthology.misc.health.smithi117.stderr: 2024-05-15T12:42:42.641 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.648+0000 7f8ab0301640 10 monclient: shutdown 2024-05-15T12:42:42.641 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.648+0000 7f8ab0301640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:42:42.641 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.648+0000 7f8ab0301640 1 -- 172.21.15.117:0/3075535206 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8aa80b1d10 msgr2=0x7f8aa80b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:42.641 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.648+0000 7f8ab0301640 1 --2- 172.21.15.117:0/3075535206 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8aa80b1d10 0x7f8aa80b21a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f8aa806cf10 tx=0x7f8a9c02b840 comp rx=0 tx=0).stop 2024-05-15T12:42:42.642 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.648+0000 7f8ab0301640 1 -- 172.21.15.117:0/3075535206 shutdown_connections 2024-05-15T12:42:42.642 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.648+0000 7f8ab0301640 1 --2- 172.21.15.117:0/3075535206 >> v2:172.21.15.178:3300/0 conn(0x7f8aa80afd80 0x7f8aa80b01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:42.642 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.648+0000 7f8ab0301640 1 --2- 172.21.15.117:0/3075535206 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8aa80b1d10 0x7f8aa80b21a0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:42.642 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.648+0000 7f8ab0301640 1 -- 172.21.15.117:0/3075535206 >> 172.21.15.117:0/3075535206 conn(0x7f8aa806d3a0 msgr2=0x7f8aa806d7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:42:42.642 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.648+0000 7f8ab0301640 1 -- 172.21.15.117:0/3075535206 shutdown_connections 2024-05-15T12:42:42.642 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.648+0000 7f8ab0301640 1 -- 172.21.15.117:0/3075535206 wait complete. 2024-05-15T12:42:42.645 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.649+0000 7f8ab0301640 10 monclient: build_initial_monmap 2024-05-15T12:42:42.645 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.649+0000 7f8ab0301640 10 monclient: monmap: 2024-05-15T12:42:42.646 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-05-15T12:42:42.646 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:42:42.646 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-05-15T12:42:42.646 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-05-15T12:42:42.646 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:42:42.646 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-05-15T12:42:42.646 INFO:teuthology.misc.health.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T12:42:42.646 INFO:teuthology.misc.health.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T12:42:42.646 INFO:teuthology.misc.health.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T12:42:42.646 INFO:teuthology.misc.health.smithi117.stderr: 2024-05-15T12:42:42.646 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.650+0000 7f8ab0301640 1 librados: starting msgr at 2024-05-15T12:42:42.646 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.650+0000 7f8ab0301640 1 librados: starting objecter 2024-05-15T12:42:42.647 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.650+0000 7f8ab0301640 1 Processor -- start 2024-05-15T12:42:42.647 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.650+0000 7f8ab0301640 1 -- start start 2024-05-15T12:42:42.647 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.650+0000 7f8ab0301640 1 librados: setting wanted keys 2024-05-15T12:42:42.647 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.650+0000 7f8ab0301640 1 librados: calling monclient init 2024-05-15T12:42:42.647 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.650+0000 7f8ab0301640 10 monclient: init 2024-05-15T12:42:42.647 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.651+0000 7f8ab0301640 10 monclient: _reopen_session rank -1 2024-05-15T12:42:42.647 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.651+0000 7f8ab0301640 10 monclient: _add_conns ranks=[0,2,1] 2024-05-15T12:42:42.647 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.651+0000 7f8ab0301640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8aa80afd80 0x7f8aa806f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:42.647 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.651+0000 7f8ab0301640 10 monclient(hunting): picked mon.noname-a con 0x7f8aa80afd80 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T12:42:42.647 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.651+0000 7f8ab0301640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8aa80b1d10 0x7f8aa8146090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:42.647 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.651+0000 7f8ab0301640 10 monclient(hunting): picked mon.noname-c con 0x7f8aa80b1d10 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:42:42.647 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.651+0000 7f8ab0301640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8aa81431e0 0x7f8aa81465d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:42.648 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.651+0000 7f8ab0301640 10 monclient(hunting): picked mon.noname-b con 0x7f8aa81431e0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:42:42.648 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.651+0000 7f8ab0301640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:42.648 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.651+0000 7f8ab0301640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa806e1b0 con 0x7f8aa80afd80 2024-05-15T12:42:42.648 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.651+0000 7f8ab0301640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:42.648 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.651+0000 7f8ab0301640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa806e030 con 0x7f8aa81431e0 2024-05-15T12:42:42.648 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.651+0000 7f8ab0301640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:42.648 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.651+0000 7f8ab0301640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa806e330 con 0x7f8aa80b1d10 2024-05-15T12:42:42.648 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.651+0000 7f8ab0301640 10 monclient(hunting): _renew_subs 2024-05-15T12:42:42.648 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.651+0000 7f8ab0301640 10 monclient(hunting): authenticate will time out at 1325.893269s 2024-05-15T12:42:42.648 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.651+0000 7f8aad89c640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8aa80b1d10 0x7f8aa8146090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:42.648 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.652+0000 7f8aae09d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8aa80afd80 0x7f8aa806f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:42.648 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.652+0000 7f8aad89c640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8aa80b1d10 0x7f8aa8146090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.117:37978/0 (socket says 172.21.15.117:37978) 2024-05-15T12:42:42.648 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.652+0000 7f8aad89c640 1 -- 172.21.15.117:0/2543767456 learned_addr learned my addr 172.21.15.117:0/2543767456 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:42:42.649 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.652+0000 7f8aad89c640 10 monclient(hunting): get_auth_request con 0x7f8aa80b1d10 auth_method 0 2024-05-15T12:42:42.649 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.652+0000 7f8aad89c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:42.649 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.652+0000 7f8aad89c640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:42.649 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.652+0000 7f8aad89c640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:42.649 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.652+0000 7f8aae09d640 10 monclient(hunting): get_auth_request con 0x7f8aa80afd80 auth_method 0 2024-05-15T12:42:42.649 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.652+0000 7f8aae09d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:42.649 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.652+0000 7f8aae09d640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:42.649 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.652+0000 7f8aae09d640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:42.649 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.653+0000 7f8aae89e640 1 --2- 172.21.15.117:0/2543767456 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8aa81431e0 0x7f8aa81465d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:42.649 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.653+0000 7f8aae89e640 10 monclient(hunting): get_auth_request con 0x7f8aa81431e0 auth_method 0 2024-05-15T12:42:42.649 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.653+0000 7f8aae89e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:42.649 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.653+0000 7f8aae89e640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:42.649 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.653+0000 7f8aae89e640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:42.649 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.653+0000 7f8aad89c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:42.650 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.653+0000 7f8aad89c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:42.650 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.653+0000 7f8aad89c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:42.650 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.653+0000 7f8aae09d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:42.650 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.653+0000 7f8aae09d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:42.650 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.653+0000 7f8aae09d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:42.650 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.653+0000 7f8aae89e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:42.650 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.653+0000 7f8aae89e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:42.650 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.653+0000 7f8aae89e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:42.652 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.656+0000 7f8aad89c640 10 monclient(hunting): handle_auth_done global_id 4187 payload 824 2024-05-15T12:42:42.652 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.656+0000 7f8aad89c640 1 -- 172.21.15.117:0/2543767456 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8aa81431e0 msgr2=0x7f8aa81465d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:42.652 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.656+0000 7f8aad89c640 1 --2- 172.21.15.117:0/2543767456 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8aa81431e0 0x7f8aa81465d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:42.652 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.656+0000 7f8aad89c640 1 -- 172.21.15.117:0/2543767456 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8aa80afd80 msgr2=0x7f8aa806f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:42.652 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.656+0000 7f8aad89c640 1 --2- 172.21.15.117:0/2543767456 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8aa80afd80 0x7f8aa806f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:42.652 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.656+0000 7f8aad89c640 10 monclient: _finish_hunting 0 2024-05-15T12:42:42.652 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.656+0000 7f8aad89c640 1 monclient: found mon.noname-c 2024-05-15T12:42:42.652 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.656+0000 7f8aad89c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:42:42.652 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.656+0000 7f8aad89c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.117:3301/0 2024-05-15T12:42:42.652 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.656+0000 7f8aad89c640 1 -- 172.21.15.117:0/2543767456 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aa8146b10 con 0x7f8aa80b1d10 2024-05-15T12:42:42.652 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.659+0000 7f8aad89c640 1 --2- 172.21.15.117:0/2543767456 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8aa80b1d10 0x7f8aa8146090 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c009890 tx=0x7f8a9c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:42:42.652 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.659+0000 7f8a9affd640 10 client.?.objecter ms_handle_connect 0x7f8aa80b1d10 2024-05-15T12:42:42.653 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.659+0000 7f8a9affd640 10 client.?.objecter resend_mon_ops 2024-05-15T12:42:42.655 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.663+0000 7f8a9affd640 1 -- 172.21.15.117:0/2543767456 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a9c032840 con 0x7f8aa80b1d10 2024-05-15T12:42:42.655 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.663+0000 7f8a9affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:42.656 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.663+0000 7f8a9affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:42:42.656 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.663+0000 7f8a9affd640 10 monclient: dump: 2024-05-15T12:42:42.656 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-05-15T12:42:42.656 INFO:teuthology.misc.health.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:42.656 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:42.656 INFO:teuthology.misc.health.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:42.656 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:42.656 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-05-15T12:42:42.656 INFO:teuthology.misc.health.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:42.656 INFO:teuthology.misc.health.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:42.656 INFO:teuthology.misc.health.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:42.656 INFO:teuthology.misc.health.smithi117.stderr: 2024-05-15T12:42:42.656 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.663+0000 7f8a9affd640 10 monclient: _finish_auth 0 2024-05-15T12:42:42.656 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.663+0000 7f8a9affd640 10 monclient: _check_auth_tickets 2024-05-15T12:42:42.656 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.663+0000 7f8a9affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:42:42.657 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.663+0000 7f8a9affd640 1 -- 172.21.15.117:0/2543767456 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a9c0329e0 con 0x7f8aa80b1d10 2024-05-15T12:42:42.657 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.663+0000 7f8a9affd640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:42:42.657 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.663+0000 7f8a9affd640 1 -- 172.21.15.117:0/2543767456 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a9c032dd0 con 0x7f8aa80b1d10 2024-05-15T12:42:42.657 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.663+0000 7f8ab0301640 5 monclient: authenticate success, global_id 4187 2024-05-15T12:42:42.657 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.663+0000 7f8a9affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:42.657 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.663+0000 7f8a9affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T12:42:42.657 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.663+0000 7f8aac89a640 4 set_mon_vals no callback set 2024-05-15T12:42:42.657 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.663+0000 7f8a9affd640 10 monclient: dump: 2024-05-15T12:42:42.657 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-05-15T12:42:42.657 INFO:teuthology.misc.health.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:42.657 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:42.657 INFO:teuthology.misc.health.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:42.657 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:42.657 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-05-15T12:42:42.657 INFO:teuthology.misc.health.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:42.657 INFO:teuthology.misc.health.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:42.658 INFO:teuthology.misc.health.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:42.658 INFO:teuthology.misc.health.smithi117.stderr: 2024-05-15T12:42:42.658 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.663+0000 7f8ab0301640 10 monclient: _renew_subs 2024-05-15T12:42:42.658 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.663+0000 7f8ab0301640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T12:42:42.658 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.663+0000 7f8ab0301640 1 -- 172.21.15.117:0/2543767456 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8aa8071130 con 0x7f8aa80b1d10 2024-05-15T12:42:42.658 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.663+0000 7f8ab0301640 10 client.4187.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:42:42.658 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.663+0000 7f8ab0301640 10 monclient: _renew_subs 2024-05-15T12:42:42.658 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.663+0000 7f8ab0301640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T12:42:42.658 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.663+0000 7f8ab0301640 1 -- 172.21.15.117:0/2543767456 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8aa8071410 con 0x7f8aa80b1d10 2024-05-15T12:42:42.658 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.663+0000 7f8ab0301640 1 librados: init done 2024-05-15T12:42:42.658 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.665+0000 7f8ab0301640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:42:42.658 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.665+0000 7f8ab0301640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:42:42.658 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.665+0000 7f8ab0301640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T12:42:42.659 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.665+0000 7f8ab0301640 1 -- 172.21.15.117:0/2543767456 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8aa804f0e0 con 0x7f8aa80b1d10 2024-05-15T12:42:42.937 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.947+0000 7f8a9affd640 1 -- 172.21.15.117:0/2543767456 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 4) v1 ==== 89722+0+0 (secure 0 0 0) 0x7f8a9c04d3e0 con 0x7f8aa80b1d10 2024-05-15T12:42:42.938 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.948+0000 7f8a9affd640 1 --2- 172.21.15.117:0/2543767456 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f8a8c06ce20 0x7f8a8c06f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:42.938 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.948+0000 7f8aae09d640 1 --2- 172.21.15.117:0/2543767456 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f8a8c06ce20 0x7f8a8c06f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:42.938 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.948+0000 7f8aae09d640 10 monclient: get_auth_request con 0x7f8a8c06ce20 auth_method 0 2024-05-15T12:42:42.938 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.948+0000 7f8a9affd640 10 client.4187.objecter ms_handle_connect 0x7f8a8c06ce20 2024-05-15T12:42:42.941 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.948+0000 7f8aae09d640 1 --2- 172.21.15.117:0/2543767456 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f8a8c06ce20 0x7f8a8c06f2e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8aa4005880 tx=0x7f8aa4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:42:42.961 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.970+0000 7f8a9affd640 1 -- 172.21.15.117:0/2543767456 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f8a9c02f030 con 0x7f8aa80b1d10 2024-05-15T12:42:42.961 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.970+0000 7f8a9affd640 10 client.4187.objecter ms_dispatch 0x7f8aa8000b90 osd_map(10..10 src has 1..10) v4 2024-05-15T12:42:42.961 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.970+0000 7f8a9affd640 3 client.4187.objecter handle_osd_map got epochs [10,10] > 0 2024-05-15T12:42:42.961 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.970+0000 7f8a9affd640 3 client.4187.objecter handle_osd_map decoding full epoch 10 2024-05-15T12:42:42.961 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.970+0000 7f8a9affd640 20 client.4187.objecter dump_active .. 0 homeless 2024-05-15T12:42:42.961 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.970+0000 7f8a9affd640 1 -- 172.21.15.117:0/2543767456 <== mon.2 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f8a9c040070 con 0x7f8aa80b1d10 2024-05-15T12:42:42.962 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.970+0000 7f8a9affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:42:42.962 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:42.970+0000 7f8a9affd640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:42:43.227 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:43.237+0000 7f8ab0301640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:42:43.227 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:43.237+0000 7f8ab0301640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:42:43.227 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:43.237+0000 7f8ab0301640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T12:42:43.227 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:43.237+0000 7f8ab0301640 1 -- 172.21.15.117:0/2543767456 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8aa8067f50 con 0x7f8aa80b1d10 2024-05-15T12:42:43.328 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:43.338+0000 7f8a9affd640 1 -- 172.21.15.117:0/2543767456 <== mon.2 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f8a9c07abc0 con 0x7f8aa80b1d10 2024-05-15T12:42:43.328 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:43.338+0000 7f8a9affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:42:43.328 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:43.338+0000 7f8a9affd640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:42:43.329 INFO:teuthology.misc.health.smithi117.stdout: 2024-05-15T12:42:43.329 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"8817f6cf-d933-4825-b972-575ef59f5ba9","created":"2024-05-15T12:42:04.264009+0000","modified":"2024-05-15T12:42:09.879357+0000","last_up_change":"0.000000","last_in_change":"2024-05-15T12:42:09.879357+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"aaffd6d9-1a00-4793-9d1f-65a30501bec4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"efd4e8ed-699c-4ccb-b989-cfadc7aaed35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"708e4e3e-7e79-480f-9dbf-7b972c2f2811","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e7b5d7d7-515c-42d9-9c41-b1ca3977865e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa647c2c-942b-4be3-afb9-6dba6b0a8cf3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5f9bc37-4bd6-4643-8f0d-e53c5f2c8782","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d5af075d-3eaa-4cb5-b6e9-6f01c201f2ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"323233c7-a3af-4fb0-ac70-de523a279b44","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-15T12:42:43.331 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:43.340+0000 7f8ab0301640 10 librados: watch_flush enter 2024-05-15T12:42:43.331 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:43.340+0000 7f8ab0301640 10 librados: watch_flush exit 2024-05-15T12:42:43.331 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:43.340+0000 7f8ab0301640 20 client.4187.objecter shutdown clearing up homeless session... 2024-05-15T12:42:43.331 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:43.340+0000 7f8ab0301640 10 client.4187.objecter successfully canceled tick 2024-05-15T12:42:43.331 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:43.340+0000 7f8ab0301640 1 -- 172.21.15.117:0/2543767456 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f8a8c06ce20 msgr2=0x7f8a8c06f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:43.331 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:43.340+0000 7f8ab0301640 1 --2- 172.21.15.117:0/2543767456 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f8a8c06ce20 0x7f8a8c06f2e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8aa4005880 tx=0x7f8aa4009040 comp rx=0 tx=0).stop 2024-05-15T12:42:43.331 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:43.340+0000 7f8ab0301640 10 monclient: shutdown 2024-05-15T12:42:43.331 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:43.340+0000 7f8ab0301640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:42:43.332 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:43.340+0000 7f8ab0301640 1 -- 172.21.15.117:0/2543767456 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8aa80b1d10 msgr2=0x7f8aa8146090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:43.332 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:43.340+0000 7f8ab0301640 1 --2- 172.21.15.117:0/2543767456 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8aa80b1d10 0x7f8aa8146090 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c009890 tx=0x7f8a9c0046c0 comp rx=0 tx=0).stop 2024-05-15T12:42:43.332 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:43.341+0000 7f8ab0301640 1 -- 172.21.15.117:0/2543767456 shutdown_connections 2024-05-15T12:42:43.332 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:43.341+0000 7f8ab0301640 1 --2- 172.21.15.117:0/2543767456 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f8a8c06ce20 0x7f8a8c06f2e0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:43.332 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:43.341+0000 7f8ab0301640 1 --2- 172.21.15.117:0/2543767456 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8aa81431e0 0x7f8aa81465d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:43.332 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:43.341+0000 7f8ab0301640 1 --2- 172.21.15.117:0/2543767456 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8aa80b1d10 0x7f8aa8146090 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:43.332 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:43.341+0000 7f8ab0301640 1 --2- 172.21.15.117:0/2543767456 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8aa80afd80 0x7f8aa806f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:43.332 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:43.341+0000 7f8ab0301640 1 -- 172.21.15.117:0/2543767456 >> 172.21.15.117:0/2543767456 conn(0x7f8aa806d3a0 msgr2=0x7f8aa8074610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:42:43.332 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:43.341+0000 7f8ab0301640 1 -- 172.21.15.117:0/2543767456 shutdown_connections 2024-05-15T12:42:43.332 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:43.341+0000 7f8ab0301640 1 -- 172.21.15.117:0/2543767456 wait complete. 2024-05-15T12:42:43.332 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:43.341+0000 7f8ab0301640 1 librados: shutdown 2024-05-15T12:42:43.342 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-05-15T12:42:45.235 INFO:tasks.ceph.osd.5.smithi178.stderr:2024-05-15T12:42:45.242+0000 5dce500 -1 osd.5 0 log_to_monitors true 2024-05-15T12:42:45.551 INFO:tasks.ceph.osd.7.smithi178.stderr:2024-05-15T12:42:45.558+0000 5dce500 -1 osd.7 0 log_to_monitors true 2024-05-15T12:42:45.675 INFO:tasks.ceph.osd.4.smithi178.stderr:2024-05-15T12:42:45.682+0000 5dce500 -1 osd.4 0 log_to_monitors true 2024-05-15T12:42:45.891 INFO:tasks.ceph.osd.6.smithi178.stderr:2024-05-15T12:42:45.893+0000 5dce500 -1 osd.6 0 log_to_monitors true 2024-05-15T12:42:46.155 INFO:tasks.ceph.osd.3.smithi117.stderr:2024-05-15T12:42:46.162+0000 5dce500 -1 osd.3 0 log_to_monitors true 2024-05-15T12:42:46.702 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-05-15T12:42:46.708+0000 5dce500 -1 osd.2 0 log_to_monitors true 2024-05-15T12:42:46.704 INFO:tasks.ceph.osd.0.smithi117.stderr:2024-05-15T12:42:46.711+0000 5dce500 -1 osd.0 0 log_to_monitors true 2024-05-15T12:42:46.908 INFO:tasks.ceph.osd.1.smithi117.stderr:2024-05-15T12:42:46.914+0000 5dce500 -1 osd.1 0 log_to_monitors true 2024-05-15T12:42:48.781 INFO:tasks.ceph.osd.7.smithi178.stderr:2024-05-15T12:42:48.787+0000 c49a640 -1 osd.7 0 waiting for initial osdmap 2024-05-15T12:42:48.785 INFO:tasks.ceph.osd.5.smithi178.stderr:2024-05-15T12:42:48.791+0000 ac84640 -1 osd.5 0 waiting for initial osdmap 2024-05-15T12:42:49.220 INFO:tasks.ceph.osd.7.smithi178.stderr:2024-05-15T12:42:49.216+0000 10260640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-15T12:42:49.244 INFO:tasks.ceph.osd.5.smithi178.stderr:2024-05-15T12:42:49.240+0000 10260640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-15T12:42:49.823 INFO:tasks.ceph.osd.3.smithi117.stderr:2024-05-15T12:42:49.829+0000 ac84640 -1 osd.3 0 waiting for initial osdmap 2024-05-15T12:42:50.149 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-15T12:42:50.268 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.278+0000 7f1f4aaff640 10 monclient: get_monmap_and_config 2024-05-15T12:42:50.268 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.278+0000 7f1f4aaff640 10 monclient: build_initial_monmap 2024-05-15T12:42:50.268 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.278+0000 7f1f4aaff640 10 monclient: monmap: 2024-05-15T12:42:50.268 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-05-15T12:42:50.268 INFO:teuthology.misc.health.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:50.268 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-05-15T12:42:50.279230+0000 2024-05-15T12:42:50.268 INFO:teuthology.misc.health.smithi117.stderr:created 2024-05-15T12:42:50.279230+0000 2024-05-15T12:42:50.268 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:42:50.269 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-05-15T12:42:50.269 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T12:42:50.269 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T12:42:50.269 INFO:teuthology.misc.health.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T12:42:50.269 INFO:teuthology.misc.health.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T12:42:50.269 INFO:teuthology.misc.health.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T12:42:50.269 INFO:teuthology.misc.health.smithi117.stderr: 2024-05-15T12:42:50.273 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.278+0000 7f1f4aaff640 1 Processor -- start 2024-05-15T12:42:50.275 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.279+0000 7f1f4aaff640 1 -- start start 2024-05-15T12:42:50.275 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.279+0000 7f1f4aaff640 10 monclient: init 2024-05-15T12:42:50.275 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.279+0000 7f1f4aaff640 10 monclient: _reopen_session rank -1 2024-05-15T12:42:50.276 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.279+0000 7f1f4aaff640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-05-15T12:42:50.276 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.279+0000 7f1f4aaff640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1f441431e0 0x7f1f440b5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:50.276 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.279+0000 7f1f4aaff640 10 monclient(hunting): picked mon.noname-b con 0x7f1f441431e0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:42:50.276 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.279+0000 7f1f4aaff640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f1f440b1d10 0x7f1f440b2170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:50.276 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.279+0000 7f1f4aaff640 10 monclient(hunting): picked mon.noname-a con 0x7f1f440b1d10 addr v2:172.21.15.117:3300/0 2024-05-15T12:42:50.276 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.279+0000 7f1f4aaff640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f1f440afd80 0x7f1f440b0210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:50.277 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.279+0000 7f1f4aaff640 10 monclient(hunting): picked mon.noname-c con 0x7f1f440afd80 addr v2:172.21.15.178:3300/0 2024-05-15T12:42:50.277 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.279+0000 7f1f4aaff640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:50.277 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.279+0000 7f1f4aaff640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f440b2840 con 0x7f1f440b1d10 2024-05-15T12:42:50.277 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.279+0000 7f1f4aaff640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:50.277 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.279+0000 7f1f4aaff640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f440b0750 con 0x7f1f440afd80 2024-05-15T12:42:50.277 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.279+0000 7f1f4aaff640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:50.278 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.279+0000 7f1f4aaff640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f440b18f0 con 0x7f1f441431e0 2024-05-15T12:42:50.278 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.279+0000 7f1f4aaff640 10 monclient(hunting): _renew_subs 2024-05-15T12:42:50.278 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.279+0000 7f1f4aaff640 10 monclient(hunting): authenticate will time out at 1333.521761s 2024-05-15T12:42:50.278 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.280+0000 7f1f4909c640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f1f440afd80 0x7f1f440b0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:50.278 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.280+0000 7f1f43fff640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f1f440b1d10 0x7f1f440b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:50.279 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.280+0000 7f1f4889b640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1f441431e0 0x7f1f440b5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:50.279 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.280+0000 7f1f4909c640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f1f440afd80 0x7f1f440b0210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:53312/0 (socket says 172.21.15.117:53312) 2024-05-15T12:42:50.279 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.280+0000 7f1f4909c640 1 -- 172.21.15.117:0/241191786 learned_addr learned my addr 172.21.15.117:0/241191786 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:42:50.279 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.280+0000 7f1f4909c640 10 monclient(hunting): get_auth_request con 0x7f1f440afd80 auth_method 0 2024-05-15T12:42:50.279 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.280+0000 7f1f4909c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:50.280 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.280+0000 7f1f4909c640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:50.280 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.280+0000 7f1f4909c640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:50.280 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.280+0000 7f1f43fff640 10 monclient(hunting): get_auth_request con 0x7f1f440b1d10 auth_method 0 2024-05-15T12:42:50.280 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.280+0000 7f1f43fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:50.280 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.280+0000 7f1f43fff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:50.280 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.280+0000 7f1f43fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:50.280 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.281+0000 7f1f4889b640 10 monclient(hunting): get_auth_request con 0x7f1f441431e0 auth_method 0 2024-05-15T12:42:50.280 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.281+0000 7f1f4889b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:50.281 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.281+0000 7f1f4889b640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:50.281 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.281+0000 7f1f4889b640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:50.281 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.281+0000 7f1f4909c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:50.281 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.281+0000 7f1f4909c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:50.281 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.281+0000 7f1f4909c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:50.282 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.281+0000 7f1f43fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:50.282 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.281+0000 7f1f43fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:50.282 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.281+0000 7f1f43fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:50.282 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.281+0000 7f1f4889b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:50.282 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.281+0000 7f1f4889b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:50.282 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.281+0000 7f1f4889b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:50.282 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.282+0000 7f1f4909c640 10 monclient(hunting): handle_auth_done global_id 4261 payload 274 2024-05-15T12:42:50.283 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.282+0000 7f1f4909c640 1 -- 172.21.15.117:0/241191786 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1f441431e0 msgr2=0x7f1f440b5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:42:50.283 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.282+0000 7f1f4909c640 1 --2- 172.21.15.117:0/241191786 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1f441431e0 0x7f1f440b5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:50.283 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.282+0000 7f1f4909c640 1 -- 172.21.15.117:0/241191786 >> v2:172.21.15.117:3300/0 conn(0x7f1f440b1d10 msgr2=0x7f1f440b2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:42:50.283 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.282+0000 7f1f4909c640 1 --2- 172.21.15.117:0/241191786 >> v2:172.21.15.117:3300/0 conn(0x7f1f440b1d10 0x7f1f440b2170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:50.283 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.283+0000 7f1f4909c640 10 monclient: _finish_hunting 0 2024-05-15T12:42:50.283 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.283+0000 7f1f4909c640 1 monclient: found mon.noname-c 2024-05-15T12:42:50.284 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.283+0000 7f1f4909c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:42:50.284 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.283+0000 7f1f4909c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.178:3300/0 2024-05-15T12:42:50.284 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.283+0000 7f1f4909c640 1 -- 172.21.15.117:0/241191786 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f440b5760 con 0x7f1f440afd80 2024-05-15T12:42:50.297 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.285+0000 7f1f4909c640 1 --2- 172.21.15.117:0/241191786 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1f440afd80 0x7f1f440b0210 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1f440aeeb0 tx=0x7f1f3c02be50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0ca9e2f4986cac4 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:42:50.297 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.288+0000 7f1f437fe640 1 -- 172.21.15.117:0/241191786 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f3c02c9f0 con 0x7f1f440afd80 2024-05-15T12:42:50.297 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.288+0000 7f1f437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:50.297 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.288+0000 7f1f437fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:42:50.297 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.288+0000 7f1f437fe640 10 monclient: dump: 2024-05-15T12:42:50.297 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-05-15T12:42:50.297 INFO:teuthology.misc.health.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:50.297 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:50.298 INFO:teuthology.misc.health.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:50.298 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:50.298 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-05-15T12:42:50.298 INFO:teuthology.misc.health.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:50.298 INFO:teuthology.misc.health.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:50.298 INFO:teuthology.misc.health.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:50.298 INFO:teuthology.misc.health.smithi117.stderr: 2024-05-15T12:42:50.299 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.288+0000 7f1f437fe640 10 monclient: _finish_auth 0 2024-05-15T12:42:50.299 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.288+0000 7f1f437fe640 10 monclient: _check_auth_tickets 2024-05-15T12:42:50.299 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.288+0000 7f1f437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:42:50.299 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.288+0000 7f1f4aaff640 5 monclient: authenticate success, global_id 4261 2024-05-15T12:42:50.299 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.288+0000 7f1f4aaff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:42:50.299 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.288+0000 7f1f437fe640 1 -- 172.21.15.117:0/241191786 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f3c02cb90 con 0x7f1f440afd80 2024-05-15T12:42:50.299 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.288+0000 7f1f437fe640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:42:50.299 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.288+0000 7f1f4aaff640 10 monclient: get_monmap_and_config success 2024-05-15T12:42:50.299 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.288+0000 7f1f437fe640 1 -- 172.21.15.117:0/241191786 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f3c0343f0 con 0x7f1f440afd80 2024-05-15T12:42:50.299 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.288+0000 7f1f4aaff640 4 set_mon_vals no callback set 2024-05-15T12:42:50.299 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.288+0000 7f1f437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:50.299 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.288+0000 7f1f437fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:42:50.299 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.288+0000 7f1f437fe640 10 monclient: dump: 2024-05-15T12:42:50.299 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-05-15T12:42:50.300 INFO:teuthology.misc.health.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:50.300 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:50.300 INFO:teuthology.misc.health.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:50.300 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:50.300 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-05-15T12:42:50.300 INFO:teuthology.misc.health.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:50.300 INFO:teuthology.misc.health.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:50.300 INFO:teuthology.misc.health.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:50.300 INFO:teuthology.misc.health.smithi117.stderr: 2024-05-15T12:42:50.300 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.289+0000 7f1f4aaff640 10 monclient: shutdown 2024-05-15T12:42:50.300 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.289+0000 7f1f4aaff640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:42:50.300 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.289+0000 7f1f4aaff640 1 -- 172.21.15.117:0/241191786 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1f440afd80 msgr2=0x7f1f440b0210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:50.300 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.289+0000 7f1f4aaff640 1 --2- 172.21.15.117:0/241191786 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1f440afd80 0x7f1f440b0210 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1f440aeeb0 tx=0x7f1f3c02be50 comp rx=0 tx=0).stop 2024-05-15T12:42:50.300 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.289+0000 7f1f4aaff640 1 -- 172.21.15.117:0/241191786 shutdown_connections 2024-05-15T12:42:50.300 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.289+0000 7f1f4aaff640 1 --2- 172.21.15.117:0/241191786 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1f440afd80 0x7f1f440b0210 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:50.301 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.289+0000 7f1f4aaff640 1 --2- 172.21.15.117:0/241191786 >> v2:172.21.15.117:3300/0 conn(0x7f1f440b1d10 0x7f1f440b2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:50.301 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.289+0000 7f1f4aaff640 1 --2- 172.21.15.117:0/241191786 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1f441431e0 0x7f1f440b5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:50.301 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.289+0000 7f1f4aaff640 1 -- 172.21.15.117:0/241191786 >> 172.21.15.117:0/241191786 conn(0x7f1f4406d3a0 msgr2=0x7f1f4406d7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:42:50.301 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.289+0000 7f1f4aaff640 1 -- 172.21.15.117:0/241191786 shutdown_connections 2024-05-15T12:42:50.301 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.289+0000 7f1f4aaff640 1 -- 172.21.15.117:0/241191786 wait complete. 2024-05-15T12:42:50.301 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.290+0000 7f1f4aaff640 10 monclient: build_initial_monmap 2024-05-15T12:42:50.301 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.290+0000 7f1f4aaff640 10 monclient: monmap: 2024-05-15T12:42:50.301 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-05-15T12:42:50.301 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:42:50.301 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-05-15T12:42:50.301 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-05-15T12:42:50.301 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:42:50.301 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-05-15T12:42:50.302 INFO:teuthology.misc.health.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T12:42:50.302 INFO:teuthology.misc.health.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T12:42:50.302 INFO:teuthology.misc.health.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T12:42:50.302 INFO:teuthology.misc.health.smithi117.stderr: 2024-05-15T12:42:50.302 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.291+0000 7f1f4aaff640 1 librados: starting msgr at 2024-05-15T12:42:50.302 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.291+0000 7f1f4aaff640 1 librados: starting objecter 2024-05-15T12:42:50.302 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.291+0000 7f1f4aaff640 1 Processor -- start 2024-05-15T12:42:50.302 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.291+0000 7f1f4aaff640 1 -- start start 2024-05-15T12:42:50.302 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.291+0000 7f1f4aaff640 1 librados: setting wanted keys 2024-05-15T12:42:50.302 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.291+0000 7f1f4aaff640 1 librados: calling monclient init 2024-05-15T12:42:50.302 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.291+0000 7f1f4aaff640 10 monclient: init 2024-05-15T12:42:50.303 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.292+0000 7f1f4aaff640 10 monclient: _reopen_session rank -1 2024-05-15T12:42:50.303 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.292+0000 7f1f4aaff640 10 monclient: _add_conns ranks=[1,0,2] 2024-05-15T12:42:50.303 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.292+0000 7f1f4aaff640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1f440afd80 0x7f1f44146230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:50.303 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.292+0000 7f1f4aaff640 10 monclient(hunting): picked mon.noname-b con 0x7f1f440afd80 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:42:50.303 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.292+0000 7f1f4aaff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1f440b1d10 0x7f1f441cd5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:50.303 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.292+0000 7f1f4aaff640 10 monclient(hunting): picked mon.noname-a con 0x7f1f440b1d10 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T12:42:50.303 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.292+0000 7f1f4aaff640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1f441431e0 0x7f1f441cc350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:50.303 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.292+0000 7f1f4aaff640 10 monclient(hunting): picked mon.noname-c con 0x7f1f441431e0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:42:50.303 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.292+0000 7f1f4aaff640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:50.303 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.292+0000 7f1f4aaff640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f440b2b50 con 0x7f1f440b1d10 2024-05-15T12:42:50.303 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.292+0000 7f1f4aaff640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:50.303 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.292+0000 7f1f4aaff640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f441435c0 con 0x7f1f440afd80 2024-05-15T12:42:50.303 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.292+0000 7f1f4aaff640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:50.303 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.292+0000 7f1f4aaff640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f440b2cd0 con 0x7f1f441431e0 2024-05-15T12:42:50.304 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.292+0000 7f1f4aaff640 10 monclient(hunting): _renew_subs 2024-05-15T12:42:50.304 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.292+0000 7f1f4aaff640 10 monclient(hunting): authenticate will time out at 1333.534271s 2024-05-15T12:42:50.304 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.293+0000 7f1f43fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1f440b1d10 0x7f1f441cd5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:50.304 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.293+0000 7f1f4909c640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1f441431e0 0x7f1f441cc350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:50.304 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.293+0000 7f1f4889b640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1f440afd80 0x7f1f44146230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:50.304 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.293+0000 7f1f4909c640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1f441431e0 0x7f1f441cc350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.117:41062/0 (socket says 172.21.15.117:41062) 2024-05-15T12:42:50.304 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.293+0000 7f1f4909c640 1 -- 172.21.15.117:0/1273137823 learned_addr learned my addr 172.21.15.117:0/1273137823 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:42:50.304 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.293+0000 7f1f43fff640 10 monclient(hunting): get_auth_request con 0x7f1f440b1d10 auth_method 0 2024-05-15T12:42:50.304 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.293+0000 7f1f43fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:50.304 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.293+0000 7f1f43fff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:50.304 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.293+0000 7f1f43fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:50.304 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.293+0000 7f1f4889b640 10 monclient(hunting): get_auth_request con 0x7f1f440afd80 auth_method 0 2024-05-15T12:42:50.305 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.293+0000 7f1f4889b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:50.305 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.293+0000 7f1f4889b640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:50.305 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.293+0000 7f1f4889b640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:50.305 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.293+0000 7f1f4909c640 10 monclient(hunting): get_auth_request con 0x7f1f441431e0 auth_method 0 2024-05-15T12:42:50.305 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.293+0000 7f1f4909c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:50.305 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.293+0000 7f1f4909c640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:50.305 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.293+0000 7f1f4909c640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:50.306 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.294+0000 7f1f4889b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:50.306 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.294+0000 7f1f4889b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:50.306 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.294+0000 7f1f4889b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:50.306 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.294+0000 7f1f43fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:50.306 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.294+0000 7f1f43fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:50.306 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.294+0000 7f1f43fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:50.306 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.294+0000 7f1f4909c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:50.307 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.294+0000 7f1f4909c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:50.307 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.294+0000 7f1f4909c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:50.307 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.296+0000 7f1f4889b640 10 monclient(hunting): handle_auth_done global_id 4264 payload 824 2024-05-15T12:42:50.307 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.297+0000 7f1f4889b640 1 -- 172.21.15.117:0/1273137823 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1f441431e0 msgr2=0x7f1f441cc350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:50.307 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.297+0000 7f1f4889b640 1 --2- 172.21.15.117:0/1273137823 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1f441431e0 0x7f1f441cc350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:50.307 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.297+0000 7f1f4889b640 1 -- 172.21.15.117:0/1273137823 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1f440b1d10 msgr2=0x7f1f441cd5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:50.308 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.297+0000 7f1f4889b640 1 --2- 172.21.15.117:0/1273137823 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1f440b1d10 0x7f1f441cd5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:50.308 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.297+0000 7f1f4889b640 10 monclient: _finish_hunting 0 2024-05-15T12:42:50.308 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.297+0000 7f1f4889b640 1 monclient: found mon.noname-b 2024-05-15T12:42:50.308 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.297+0000 7f1f4889b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:42:50.308 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.297+0000 7f1f4889b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2024-05-15T12:42:50.308 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.297+0000 7f1f4889b640 1 -- 172.21.15.117:0/1273137823 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f440a9720 con 0x7f1f440afd80 2024-05-15T12:42:50.308 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.299+0000 7f1f4889b640 1 --2- 172.21.15.117:0/1273137823 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1f440afd80 0x7f1f44146230 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1f3800e870 tx=0x7f1f3800ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:42:50.309 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.299+0000 7f1f417fa640 10 client.?.objecter ms_handle_connect 0x7f1f440afd80 2024-05-15T12:42:50.309 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.299+0000 7f1f417fa640 10 client.?.objecter resend_mon_ops 2024-05-15T12:42:50.309 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.303+0000 7f1f417fa640 1 -- 172.21.15.117:0/1273137823 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f38017070 con 0x7f1f440afd80 2024-05-15T12:42:50.309 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.303+0000 7f1f417fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:50.309 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.303+0000 7f1f417fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:42:50.309 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.303+0000 7f1f417fa640 10 monclient: dump: 2024-05-15T12:42:50.310 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-05-15T12:42:50.310 INFO:teuthology.misc.health.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:50.310 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:50.310 INFO:teuthology.misc.health.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:50.310 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:50.310 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-05-15T12:42:50.310 INFO:teuthology.misc.health.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:50.311 INFO:teuthology.misc.health.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:50.311 INFO:teuthology.misc.health.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:50.311 INFO:teuthology.misc.health.smithi117.stderr: 2024-05-15T12:42:50.311 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.303+0000 7f1f417fa640 10 monclient: _finish_auth 0 2024-05-15T12:42:50.311 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.303+0000 7f1f417fa640 10 monclient: _check_auth_tickets 2024-05-15T12:42:50.311 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.303+0000 7f1f417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:42:50.311 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.303+0000 7f1f417fa640 1 -- 172.21.15.117:0/1273137823 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f3800ee60 con 0x7f1f440afd80 2024-05-15T12:42:50.311 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.303+0000 7f1f4aaff640 5 monclient: authenticate success, global_id 4264 2024-05-15T12:42:50.312 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.303+0000 7f1f417fa640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:42:50.312 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.303+0000 7f1f417fa640 1 -- 172.21.15.117:0/1273137823 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f3800cb60 con 0x7f1f440afd80 2024-05-15T12:42:50.312 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.303+0000 7f1f417fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:50.312 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.303+0000 7f1f417fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:42:50.312 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.303+0000 7f1f417fa640 10 monclient: dump: 2024-05-15T12:42:50.312 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-05-15T12:42:50.312 INFO:teuthology.misc.health.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:50.313 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:50.313 INFO:teuthology.misc.health.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:50.313 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:50.313 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-05-15T12:42:50.313 INFO:teuthology.misc.health.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:50.313 INFO:teuthology.misc.health.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:50.314 INFO:teuthology.misc.health.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:50.314 INFO:teuthology.misc.health.smithi117.stderr: 2024-05-15T12:42:50.314 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.303+0000 7f1f4aaff640 10 monclient: _renew_subs 2024-05-15T12:42:50.314 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.303+0000 7f1f4aaff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:42:50.314 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.303+0000 7f1f42ffd640 4 set_mon_vals no callback set 2024-05-15T12:42:50.314 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.303+0000 7f1f4aaff640 1 -- 172.21.15.117:0/1273137823 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f441cbe30 con 0x7f1f440afd80 2024-05-15T12:42:50.315 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.303+0000 7f1f4aaff640 10 client.4264.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:42:50.315 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.303+0000 7f1f4aaff640 10 monclient: _renew_subs 2024-05-15T12:42:50.315 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.303+0000 7f1f4aaff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:42:50.315 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.303+0000 7f1f4aaff640 1 -- 172.21.15.117:0/1273137823 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f441cc890 con 0x7f1f440afd80 2024-05-15T12:42:50.315 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.303+0000 7f1f4aaff640 1 librados: init done 2024-05-15T12:42:50.315 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.304+0000 7f1f4aaff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:42:50.315 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.305+0000 7f1f4aaff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:42:50.315 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.305+0000 7f1f4aaff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:42:50.315 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.305+0000 7f1f4aaff640 1 -- 172.21.15.117:0/1273137823 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f441cbc30 con 0x7f1f440afd80 2024-05-15T12:42:50.317 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.322+0000 7f1f417fa640 1 -- 172.21.15.117:0/1273137823 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 4) v1 ==== 89722+0+0 (secure 0 0 0) 0x7f1f3801a070 con 0x7f1f440afd80 2024-05-15T12:42:50.317 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.323+0000 7f1f417fa640 1 --2- 172.21.15.117:0/1273137823 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f1f2806ce20 0x7f1f2806f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:50.317 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.323+0000 7f1f43fff640 1 --2- 172.21.15.117:0/1273137823 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f1f2806ce20 0x7f1f2806f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:50.317 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.323+0000 7f1f43fff640 10 monclient: get_auth_request con 0x7f1f2806ce20 auth_method 0 2024-05-15T12:42:50.317 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.323+0000 7f1f417fa640 10 client.4264.objecter ms_handle_connect 0x7f1f2806ce20 2024-05-15T12:42:50.317 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.323+0000 7f1f43fff640 1 --2- 172.21.15.117:0/1273137823 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f1f2806ce20 0x7f1f2806f2e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1f440ae820 tx=0x7f1f30009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:42:50.317 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.326+0000 7f1f417fa640 1 -- 172.21.15.117:0/1273137823 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 2511+0+0 (secure 0 0 0) 0x7f1f38057000 con 0x7f1f440afd80 2024-05-15T12:42:50.317 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.326+0000 7f1f417fa640 10 client.4264.objecter ms_dispatch 0x7f1f44000b90 osd_map(13..13 src has 1..13) v4 2024-05-15T12:42:50.317 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.326+0000 7f1f417fa640 3 client.4264.objecter handle_osd_map got epochs [13,13] > 0 2024-05-15T12:42:50.317 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.326+0000 7f1f417fa640 3 client.4264.objecter handle_osd_map decoding full epoch 13 2024-05-15T12:42:50.317 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.326+0000 7f1f417fa640 20 client.4264.objecter dump_active .. 0 homeless 2024-05-15T12:42:50.362 INFO:tasks.ceph.osd.3.smithi117.stderr:2024-05-15T12:42:50.358+0000 10260640 -1 osd.3 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-15T12:42:50.565 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.575+0000 7f1f417fa640 1 -- 172.21.15.117:0/1273137823 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f1f38058f60 con 0x7f1f440afd80 2024-05-15T12:42:50.565 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.575+0000 7f1f417fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:42:50.565 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.575+0000 7f1f417fa640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:42:50.805 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.815+0000 7f1f4aaff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:42:50.806 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.815+0000 7f1f4aaff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:42:50.806 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.815+0000 7f1f4aaff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:42:50.806 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.815+0000 7f1f4aaff640 1 -- 172.21.15.117:0/1273137823 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1f4404f0e0 con 0x7f1f440afd80 2024-05-15T12:42:50.825 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.835+0000 7f1f417fa640 1 -- 172.21.15.117:0/1273137823 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+8076 (secure 0 0 0) 0x7f1f3805d000 con 0x7f1f440afd80 2024-05-15T12:42:50.825 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.835+0000 7f1f417fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:42:50.825 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.835+0000 7f1f417fa640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:42:50.826 INFO:teuthology.misc.health.smithi117.stdout: 2024-05-15T12:42:50.826 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":14,"fsid":"8817f6cf-d933-4825-b972-575ef59f5ba9","created":"2024-05-15T12:42:04.264009+0000","modified":"2024-05-15T12:42:50.434944+0000","last_up_change":"2024-05-15T12:42:50.434944+0000","last_in_change":"2024-05-15T12:42:09.879357+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"aaffd6d9-1a00-4793-9d1f-65a30501bec4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"efd4e8ed-699c-4ccb-b989-cfadc7aaed35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"708e4e3e-7e79-480f-9dbf-7b972c2f2811","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e7b5d7d7-515c-42d9-9c41-b1ca3977865e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa647c2c-942b-4be3-afb9-6dba6b0a8cf3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5f9bc37-4bd6-4643-8f0d-e53c5f2c8782","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":3713764498},{"type":"v1","addr":"172.21.15.178:6817","nonce":3713764498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":3713764498},{"type":"v1","addr":"172.21.15.178:6819","nonce":3713764498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":3713764498},{"type":"v1","addr":"172.21.15.178:6823","nonce":3713764498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":3713764498},{"type":"v1","addr":"172.21.15.178:6821","nonce":3713764498}]},"public_addr":"172.21.15.178:6817/3713764498","cluster_addr":"172.21.15.178:6819/3713764498","heartbeat_back_addr":"172.21.15.178:6823/3713764498","heartbeat_front_addr":"172.21.15.178:6821/3713764498","state":["exists","up"]},{"osd":6,"uuid":"d5af075d-3eaa-4cb5-b6e9-6f01c201f2ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"323233c7-a3af-4fb0-ac70-de523a279b44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":3049776526},{"type":"v1","addr":"172.21.15.178:6809","nonce":3049776526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":3049776526},{"type":"v1","addr":"172.21.15.178:6811","nonce":3049776526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":3049776526},{"type":"v1","addr":"172.21.15.178:6815","nonce":3049776526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":3049776526},{"type":"v1","addr":"172.21.15.178:6813","nonce":3049776526}]},"public_addr":"172.21.15.178:6809/3049776526","cluster_addr":"172.21.15.178:6811/3049776526","heartbeat_back_addr":"172.21.15.178:6815/3049776526","heartbeat_front_addr":"172.21.15.178:6813/3049776526","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:42:47.675658+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-15T12:42:50.829 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.838+0000 7f1f4aaff640 10 librados: watch_flush enter 2024-05-15T12:42:50.829 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.838+0000 7f1f4aaff640 10 librados: watch_flush exit 2024-05-15T12:42:50.829 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.838+0000 7f1f4aaff640 20 client.4264.objecter shutdown clearing up homeless session... 2024-05-15T12:42:50.829 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.838+0000 7f1f4aaff640 10 client.4264.objecter successfully canceled tick 2024-05-15T12:42:50.829 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.838+0000 7f1f4aaff640 1 -- 172.21.15.117:0/1273137823 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f1f2806ce20 msgr2=0x7f1f2806f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:50.829 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.838+0000 7f1f4aaff640 1 --2- 172.21.15.117:0/1273137823 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f1f2806ce20 0x7f1f2806f2e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1f440ae820 tx=0x7f1f30009190 comp rx=0 tx=0).stop 2024-05-15T12:42:50.829 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.838+0000 7f1f4aaff640 10 monclient: shutdown 2024-05-15T12:42:50.829 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.838+0000 7f1f4aaff640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:42:50.829 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.838+0000 7f1f4aaff640 1 -- 172.21.15.117:0/1273137823 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1f440afd80 msgr2=0x7f1f44146230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:50.829 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.838+0000 7f1f4aaff640 1 --2- 172.21.15.117:0/1273137823 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1f440afd80 0x7f1f44146230 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1f3800e870 tx=0x7f1f3800ed30 comp rx=0 tx=0).stop 2024-05-15T12:42:50.829 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.838+0000 7f1f4aaff640 1 -- 172.21.15.117:0/1273137823 shutdown_connections 2024-05-15T12:42:50.829 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.838+0000 7f1f4aaff640 1 --2- 172.21.15.117:0/1273137823 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1f441431e0 0x7f1f441cc350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:50.830 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.838+0000 7f1f4aaff640 1 --2- 172.21.15.117:0/1273137823 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1f440b1d10 0x7f1f441cd5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:50.830 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.838+0000 7f1f4aaff640 1 --2- 172.21.15.117:0/1273137823 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f1f2806ce20 0x7f1f2806f2e0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:50.830 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.838+0000 7f1f4aaff640 1 --2- 172.21.15.117:0/1273137823 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1f440afd80 0x7f1f44146230 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:50.830 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.838+0000 7f1f4aaff640 1 -- 172.21.15.117:0/1273137823 >> 172.21.15.117:0/1273137823 conn(0x7f1f4406d3a0 msgr2=0x7f1f4413f7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:42:50.830 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.838+0000 7f1f4aaff640 1 -- 172.21.15.117:0/1273137823 shutdown_connections 2024-05-15T12:42:50.830 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.838+0000 7f1f4aaff640 1 -- 172.21.15.117:0/1273137823 wait complete. 2024-05-15T12:42:50.830 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:50.838+0000 7f1f4aaff640 1 librados: shutdown 2024-05-15T12:42:50.841 DEBUG:teuthology.misc:2 of 8 OSDs are up 2024-05-15T12:42:56.017 INFO:tasks.ceph.osd.6.smithi178.stderr:2024-05-15T12:42:55.859+0000 5dce500 -1 bdev(0x18adf650 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-05-15T12:42:56.017 INFO:tasks.ceph.osd.6.smithi178.stderr:2024-05-15T12:42:55.968+0000 5dce500 -1 bdev(0x18adf650 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-05-15T12:42:56.120 INFO:tasks.ceph.osd.6.smithi178.stderr:2024-05-15T12:42:56.027+0000 5dce500 -1 bdev(0x18adf650 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-05-15T12:42:56.120 INFO:tasks.ceph.osd.6.smithi178.stderr:2024-05-15T12:42:56.106+0000 5dce500 -1 bdev(0x18adf650 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-05-15T12:42:56.143 INFO:tasks.ceph.osd.4.smithi178.stderr:2024-05-15T12:42:55.873+0000 5dce500 -1 bdev(0x18e82d10 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 2 2024-05-15T12:42:56.202 INFO:tasks.ceph.osd.4.smithi178.stderr:2024-05-15T12:42:55.919+0000 5dce500 -1 bdev(0x18e82d10 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-05-15T12:42:56.202 INFO:tasks.ceph.osd.4.smithi178.stderr:2024-05-15T12:42:56.129+0000 5dce500 -1 bdev(0x18e82d10 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-05-15T12:42:56.202 INFO:tasks.ceph.osd.4.smithi178.stderr:2024-05-15T12:42:56.141+0000 5dce500 -1 bdev(0x18e82d10 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-05-15T12:42:56.346 INFO:tasks.ceph.osd.6.smithi178.stderr:2024-05-15T12:42:56.134+0000 5dce500 -1 bdev(0x18adf650 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-05-15T12:42:56.358 INFO:tasks.ceph.osd.6.smithi178.stderr:2024-05-15T12:42:56.339+0000 5dce500 -1 bdev(0x18adf650 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-05-15T12:42:56.640 INFO:tasks.ceph.osd.4.smithi178.stderr:2024-05-15T12:42:56.218+0000 5dce500 -1 bdev(0x18e82d10 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 3 2024-05-15T12:42:56.671 INFO:tasks.ceph.osd.6.smithi178.stderr:2024-05-15T12:42:56.356+0000 5dce500 -1 bdev(0x18adf650 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-05-15T12:42:56.671 INFO:tasks.ceph.osd.6.smithi178.stderr:2024-05-15T12:42:56.369+0000 5dce500 -1 bdev(0x18adf650 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-05-15T12:42:56.671 INFO:tasks.ceph.osd.6.smithi178.stderr:2024-05-15T12:42:56.381+0000 5dce500 -1 bdev(0x18adf650 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-05-15T12:42:56.672 INFO:tasks.ceph.osd.6.smithi178.stderr:2024-05-15T12:42:56.440+0000 5dce500 -1 bdev(0x18adf650 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-05-15T12:42:56.766 INFO:tasks.ceph.osd.6.smithi178.stderr:2024-05-15T12:42:56.656+0000 5dce500 -1 bdev(0x18adf650 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-05-15T12:42:56.776 INFO:tasks.ceph.osd.6.smithi178.stderr:2024-05-15T12:42:56.681+0000 5dce500 -1 bdev(0x18adf650 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-05-15T12:42:57.076 INFO:tasks.ceph.osd.1.smithi117.stderr:2024-05-15T12:42:57.057+0000 5dce500 -1 bdev(0x18e82d10 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 2 2024-05-15T12:42:57.110 INFO:tasks.ceph.osd.1.smithi117.stderr:2024-05-15T12:42:57.086+0000 5dce500 -1 bdev(0x18e82d10 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-05-15T12:42:57.110 INFO:tasks.ceph.osd.1.smithi117.stderr:2024-05-15T12:42:57.098+0000 5dce500 -1 bdev(0x18e82d10 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-05-15T12:42:57.153 INFO:tasks.ceph.osd.1.smithi117.stderr:2024-05-15T12:42:57.120+0000 5dce500 -1 bdev(0x18e82d10 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-05-15T12:42:57.194 INFO:tasks.ceph.osd.0.smithi117.stderr:2024-05-15T12:42:57.180+0000 5dce500 -1 bdev(0x18e83880 /var/lib/ceph/osd/ceph-0/block) aio_submit retries 1 2024-05-15T12:42:57.194 INFO:tasks.ceph.osd.0.smithi117.stderr:2024-05-15T12:42:57.192+0000 5dce500 -1 bdev(0x18e83880 /var/lib/ceph/osd/ceph-0/block) aio_submit retries 1 2024-05-15T12:42:57.201 INFO:tasks.ceph.osd.1.smithi117.stderr:2024-05-15T12:42:57.190+0000 5dce500 -1 bdev(0x18e82d10 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-05-15T12:42:57.208 INFO:tasks.ceph.osd.0.smithi117.stderr:2024-05-15T12:42:57.204+0000 5dce500 -1 bdev(0x18e83880 /var/lib/ceph/osd/ceph-0/block) aio_submit retries 2 2024-05-15T12:42:57.296 INFO:tasks.ceph.osd.1.smithi117.stderr:2024-05-15T12:42:57.212+0000 5dce500 -1 bdev(0x18e82d10 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-05-15T12:42:57.296 INFO:tasks.ceph.osd.1.smithi117.stderr:2024-05-15T12:42:57.293+0000 5dce500 -1 bdev(0x18e82d10 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-05-15T12:42:57.317 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-05-15T12:42:57.215+0000 5dce500 -1 bdev(0x18e82d10 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 2 2024-05-15T12:42:57.346 INFO:tasks.ceph.osd.1.smithi117.stderr:2024-05-15T12:42:57.340+0000 5dce500 -1 bdev(0x18e82d10 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 2 2024-05-15T12:42:57.406 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-05-15T12:42:57.357+0000 5dce500 -1 bdev(0x18e82d10 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-05-15T12:42:57.406 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-05-15T12:42:57.378+0000 5dce500 -1 bdev(0x18e82d10 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-05-15T12:42:57.464 INFO:tasks.ceph.osd.0.smithi117.stderr:2024-05-15T12:42:57.218+0000 5dce500 -1 bdev(0x18e83880 /var/lib/ceph/osd/ceph-0/block) aio_submit retries 1 2024-05-15T12:42:57.474 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-05-15T12:42:57.417+0000 5dce500 -1 bdev(0x18e82d10 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-05-15T12:42:57.476 INFO:tasks.ceph.osd.0.smithi117.stderr:2024-05-15T12:42:57.379+0000 5dce500 -1 bdev(0x18e83880 /var/lib/ceph/osd/ceph-0/block) aio_submit retries 1 2024-05-15T12:42:57.486 INFO:tasks.ceph.osd.1.smithi117.stderr:2024-05-15T12:42:57.366+0000 5dce500 -1 bdev(0x18e82d10 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 2 2024-05-15T12:42:57.487 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-05-15T12:42:57.428+0000 5dce500 -1 bdev(0x18e82d10 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-05-15T12:42:57.494 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-05-15T12:42:57.484+0000 5dce500 -1 bdev(0x18e82d10 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-05-15T12:42:57.494 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-05-15T12:42:57.498+0000 5dce500 -1 bdev(0x18e82d10 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-05-15T12:42:57.500 INFO:tasks.ceph.osd.0.smithi117.stderr:2024-05-15T12:42:57.475+0000 5dce500 -1 bdev(0x18e83880 /var/lib/ceph/osd/ceph-0/block) aio_submit retries 1 2024-05-15T12:42:57.511 INFO:tasks.ceph.osd.0.smithi117.stderr:2024-05-15T12:42:57.487+0000 5dce500 -1 bdev(0x18e83880 /var/lib/ceph/osd/ceph-0/block) aio_submit retries 1 2024-05-15T12:42:57.526 INFO:tasks.ceph.osd.1.smithi117.stderr:2024-05-15T12:42:57.390+0000 5dce500 -1 bdev(0x18e82d10 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 2 2024-05-15T12:42:57.536 INFO:tasks.ceph.osd.1.smithi117.stderr:2024-05-15T12:42:57.399+0000 5dce500 -1 bdev(0x18e82d10 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-05-15T12:42:57.536 INFO:tasks.ceph.osd.1.smithi117.stderr:2024-05-15T12:42:57.445+0000 5dce500 -1 bdev(0x18e82d10 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-05-15T12:42:57.537 INFO:tasks.ceph.osd.1.smithi117.stderr:2024-05-15T12:42:57.536+0000 5dce500 -1 bdev(0x18e82d10 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 2 2024-05-15T12:42:57.623 INFO:tasks.ceph.osd.0.smithi117.stderr:2024-05-15T12:42:57.511+0000 5dce500 -1 bdev(0x18e83880 /var/lib/ceph/osd/ceph-0/block) aio_submit retries 1 2024-05-15T12:42:57.642 INFO:tasks.ceph.osd.0.smithi117.stderr:2024-05-15T12:42:57.633+0000 5dce500 -1 bdev(0x18e83880 /var/lib/ceph/osd/ceph-0/block) aio_submit retries 1 2024-05-15T12:42:57.646 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-15T12:42:57.720 INFO:tasks.ceph.osd.0.smithi117.stderr:2024-05-15T12:42:57.652+0000 5dce500 -1 bdev(0x18e83880 /var/lib/ceph/osd/ceph-0/block) aio_submit retries 1 2024-05-15T12:42:57.725 INFO:tasks.ceph.osd.6.smithi178.stderr:2024-05-15T12:42:57.731+0000 c49a640 -1 osd.6 0 waiting for initial osdmap 2024-05-15T12:42:57.751 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.761+0000 7f2fcd4cf640 10 monclient: get_monmap_and_config 2024-05-15T12:42:57.751 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.761+0000 7f2fcd4cf640 10 monclient: build_initial_monmap 2024-05-15T12:42:57.751 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.761+0000 7f2fcd4cf640 10 monclient: monmap: 2024-05-15T12:42:57.751 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-05-15T12:42:57.751 INFO:teuthology.misc.health.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:57.751 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-05-15T12:42:57.762274+0000 2024-05-15T12:42:57.751 INFO:teuthology.misc.health.smithi117.stderr:created 2024-05-15T12:42:57.762274+0000 2024-05-15T12:42:57.751 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:42:57.752 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-05-15T12:42:57.752 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T12:42:57.752 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T12:42:57.752 INFO:teuthology.misc.health.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T12:42:57.752 INFO:teuthology.misc.health.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T12:42:57.752 INFO:teuthology.misc.health.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T12:42:57.752 INFO:teuthology.misc.health.smithi117.stderr: 2024-05-15T12:42:57.754 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.761+0000 7f2fcd4cf640 1 Processor -- start 2024-05-15T12:42:57.754 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.762+0000 7f2fcd4cf640 1 -- start start 2024-05-15T12:42:57.754 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.762+0000 7f2fcd4cf640 10 monclient: init 2024-05-15T12:42:57.754 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.762+0000 7f2fcd4cf640 10 monclient: _reopen_session rank -1 2024-05-15T12:42:57.754 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.762+0000 7f2fcd4cf640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-05-15T12:42:57.755 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.762+0000 7f2fcd4cf640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2fc80b54b0 0x7f2fc80a9980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:57.755 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.762+0000 7f2fcd4cf640 10 monclient(hunting): picked mon.noname-b con 0x7f2fc80b54b0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:42:57.755 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.762+0000 7f2fcd4cf640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2fc81426a0 addr v1:172.21.15.178:6789/0 2024-05-15T12:42:57.755 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.762+0000 7f2fcd4cf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2fc813f850 addr v1:172.21.15.117:6789/0 2024-05-15T12:42:57.755 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.762+0000 7f2fcd4cf640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fc80ab770 con 0x7f2fc813f850 2024-05-15T12:42:57.755 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.762+0000 7f2fcd4cf640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fc80aa050 con 0x7f2fc81426a0 2024-05-15T12:42:57.755 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.762+0000 7f2fcd4cf640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:57.755 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.762+0000 7f2fcd4cf640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc80b5890 con 0x7f2fc80b54b0 2024-05-15T12:42:57.755 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.762+0000 7f2fcd4cf640 10 monclient(hunting): _renew_subs 2024-05-15T12:42:57.755 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.762+0000 7f2fcd4cf640 10 monclient(hunting): authenticate will time out at 1341.004465s 2024-05-15T12:42:57.755 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.763+0000 7f2fc6ffd640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2fc80b54b0 0x7f2fc80a9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:57.755 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.763+0000 7f2fc67fc640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f2fc81426a0 0x7f2fc8142ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.117:56594/0 (socket says 172.21.15.117:56594) 2024-05-15T12:42:57.756 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.763+0000 7f2fc67fc640 1 -- 172.21.15.117:0/672251155 learned_addr learned my addr 172.21.15.117:0/672251155 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-05-15T12:42:57.756 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.763+0000 7f2fc6ffd640 10 monclient(hunting): get_auth_request con 0x7f2fc80b54b0 auth_method 0 2024-05-15T12:42:57.756 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.763+0000 7f2fc6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:57.756 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.763+0000 7f2fc6ffd640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:57.756 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.763+0000 7f2fc6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:57.756 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.764+0000 7f2fc6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:57.756 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.764+0000 7f2fc6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:57.756 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.764+0000 7f2fc6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:57.757 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.765+0000 7f2fc5ffb640 1 -- 172.21.15.117:0/672251155 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3643854760 0 0) 0x7f2fc80aa050 con 0x7f2fc81426a0 2024-05-15T12:42:57.757 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.765+0000 7f2fc5ffb640 10 monclient(hunting): _negotiate 2024-05-15T12:42:57.757 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.765+0000 7f2fc5ffb640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:57.757 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.765+0000 7f2fc5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:57.758 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.765+0000 7f2fc5ffb640 10 monclient(hunting): my global_id is 4273 2024-05-15T12:42:57.758 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.765+0000 7f2fc5ffb640 1 -- 172.21.15.117:0/672251155 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fb0003520 con 0x7f2fc81426a0 2024-05-15T12:42:57.758 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.767+0000 7f2fc6ffd640 10 monclient(hunting): handle_auth_done global_id 4226 payload 274 2024-05-15T12:42:57.758 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.767+0000 7f2fc6ffd640 1 -- 172.21.15.117:0/672251155 >> v1:172.21.15.178:6789/0 conn(0x7f2fc81426a0 legacy=0x7f2fc8142ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:57.758 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.767+0000 7f2fc6ffd640 1 -- 172.21.15.117:0/672251155 >> v1:172.21.15.117:6789/0 conn(0x7f2fc813f850 legacy=0x7f2fc813fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:42:57.758 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.767+0000 7f2fc6ffd640 10 monclient: _finish_hunting 0 2024-05-15T12:42:57.758 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.767+0000 7f2fc6ffd640 1 monclient: found mon.noname-b 2024-05-15T12:42:57.758 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.767+0000 7f2fc6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:42:57.758 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.767+0000 7f2fc6ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.117:3301/0 2024-05-15T12:42:57.758 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.767+0000 7f2fc6ffd640 1 -- 172.21.15.117:0/672251155 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc80aa570 con 0x7f2fc80b54b0 2024-05-15T12:42:57.759 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.769+0000 7f2fc6ffd640 1 --2- 172.21.15.117:0/672251155 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2fc80b54b0 0x7f2fc80a9980 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2fb80097d0 tx=0x7f2fb802b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c765d88cc148373 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:42:57.765 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.775+0000 7f2fc5ffb640 1 -- 172.21.15.117:0/672251155 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb8031770 con 0x7f2fc80b54b0 2024-05-15T12:42:57.765 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.775+0000 7f2fc5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:57.766 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.775+0000 7f2fc5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:42:57.766 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.775+0000 7f2fc5ffb640 10 monclient: dump: 2024-05-15T12:42:57.766 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-05-15T12:42:57.766 INFO:teuthology.misc.health.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:57.766 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:57.766 INFO:teuthology.misc.health.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:57.766 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:57.766 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-05-15T12:42:57.766 INFO:teuthology.misc.health.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:57.766 INFO:teuthology.misc.health.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:57.766 INFO:teuthology.misc.health.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:57.766 INFO:teuthology.misc.health.smithi117.stderr: 2024-05-15T12:42:57.766 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.775+0000 7f2fc5ffb640 10 monclient: _finish_auth 0 2024-05-15T12:42:57.766 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.775+0000 7f2fc5ffb640 10 monclient: _check_auth_tickets 2024-05-15T12:42:57.767 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.775+0000 7f2fc5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:42:57.767 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.775+0000 7f2fc5ffb640 1 -- 172.21.15.117:0/672251155 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fb8037070 con 0x7f2fc80b54b0 2024-05-15T12:42:57.767 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.775+0000 7f2fcd4cf640 5 monclient: authenticate success, global_id 4226 2024-05-15T12:42:57.767 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.775+0000 7f2fcd4cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:42:57.767 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.775+0000 7f2fc5ffb640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:42:57.767 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.775+0000 7f2fc5ffb640 1 -- 172.21.15.117:0/672251155 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb8031b30 con 0x7f2fc80b54b0 2024-05-15T12:42:57.767 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.775+0000 7f2fc5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:57.767 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.775+0000 7f2fc5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T12:42:57.767 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.775+0000 7f2fc5ffb640 10 monclient: dump: 2024-05-15T12:42:57.767 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-05-15T12:42:57.767 INFO:teuthology.misc.health.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:57.767 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:57.767 INFO:teuthology.misc.health.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:57.767 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:57.767 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-05-15T12:42:57.767 INFO:teuthology.misc.health.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:57.768 INFO:teuthology.misc.health.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:57.768 INFO:teuthology.misc.health.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:57.768 INFO:teuthology.misc.health.smithi117.stderr: 2024-05-15T12:42:57.768 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.775+0000 7f2fcd4cf640 10 monclient: get_monmap_and_config success 2024-05-15T12:42:57.768 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.775+0000 7f2fcd4cf640 4 set_mon_vals no callback set 2024-05-15T12:42:57.769 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.776+0000 7f2fcd4cf640 10 monclient: shutdown 2024-05-15T12:42:57.769 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.776+0000 7f2fcd4cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:42:57.769 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.776+0000 7f2fcd4cf640 1 -- 172.21.15.117:0/672251155 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2fc80b54b0 msgr2=0x7f2fc80a9980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:57.770 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.776+0000 7f2fcd4cf640 1 --2- 172.21.15.117:0/672251155 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2fc80b54b0 0x7f2fc80a9980 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2fb80097d0 tx=0x7f2fb802b810 comp rx=0 tx=0).stop 2024-05-15T12:42:57.770 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.777+0000 7f2fcd4cf640 1 -- 172.21.15.117:0/672251155 shutdown_connections 2024-05-15T12:42:57.770 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.777+0000 7f2fcd4cf640 1 --2- 172.21.15.117:0/672251155 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2fc80b54b0 0x7f2fc80a9980 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:57.770 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.777+0000 7f2fcd4cf640 1 -- 172.21.15.117:0/672251155 >> 172.21.15.117:0/672251155 conn(0x7f2fc806d370 msgr2=0x7f2fc806d780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:42:57.770 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.777+0000 7f2fcd4cf640 1 -- 172.21.15.117:0/672251155 shutdown_connections 2024-05-15T12:42:57.770 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.778+0000 7f2fcd4cf640 1 -- 172.21.15.117:0/672251155 wait complete. 2024-05-15T12:42:57.770 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.778+0000 7f2fcd4cf640 10 monclient: build_initial_monmap 2024-05-15T12:42:57.770 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.779+0000 7f2fcd4cf640 10 monclient: monmap: 2024-05-15T12:42:57.770 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-05-15T12:42:57.770 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:42:57.770 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-05-15T12:42:57.770 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-05-15T12:42:57.770 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:42:57.770 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-05-15T12:42:57.771 INFO:teuthology.misc.health.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T12:42:57.771 INFO:teuthology.misc.health.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T12:42:57.771 INFO:teuthology.misc.health.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T12:42:57.771 INFO:teuthology.misc.health.smithi117.stderr: 2024-05-15T12:42:57.771 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.779+0000 7f2fcd4cf640 1 librados: starting msgr at 2024-05-15T12:42:57.771 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.779+0000 7f2fcd4cf640 1 librados: starting objecter 2024-05-15T12:42:57.771 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.779+0000 7f2fcd4cf640 1 Processor -- start 2024-05-15T12:42:57.771 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.779+0000 7f2fcd4cf640 1 -- start start 2024-05-15T12:42:57.771 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.779+0000 7f2fcd4cf640 1 librados: setting wanted keys 2024-05-15T12:42:57.771 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.779+0000 7f2fcd4cf640 1 librados: calling monclient init 2024-05-15T12:42:57.771 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.779+0000 7f2fcd4cf640 10 monclient: init 2024-05-15T12:42:57.774 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.780+0000 7f2fcd4cf640 10 monclient: _reopen_session rank -1 2024-05-15T12:42:57.774 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.780+0000 7f2fcd4cf640 10 monclient: _add_conns ranks=[1,0,2] 2024-05-15T12:42:57.774 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.780+0000 7f2fcd4cf640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2fc80b54b0 0x7f2fc8143c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:57.774 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.780+0000 7f2fcd4cf640 10 monclient(hunting): picked mon.noname-b con 0x7f2fc80b54b0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:42:57.774 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.780+0000 7f2fcd4cf640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2fc813f850 0x7f2fc8145890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:57.774 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.780+0000 7f2fcd4cf640 10 monclient(hunting): picked mon.noname-a con 0x7f2fc813f850 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T12:42:57.774 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.780+0000 7f2fcd4cf640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2fc81426a0 0x7f2fc8145dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:57.774 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.780+0000 7f2fcd4cf640 10 monclient(hunting): picked mon.noname-c con 0x7f2fc81426a0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:42:57.774 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.780+0000 7f2fcd4cf640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:57.774 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.780+0000 7f2fcd4cf640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc814e100 con 0x7f2fc813f850 2024-05-15T12:42:57.774 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.780+0000 7f2fcd4cf640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:57.774 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.780+0000 7f2fcd4cf640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc814df80 con 0x7f2fc80b54b0 2024-05-15T12:42:57.775 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.780+0000 7f2fcd4cf640 10 monclient(hunting): start opening mon connection 2024-05-15T12:42:57.775 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.780+0000 7f2fcd4cf640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc814e280 con 0x7f2fc81426a0 2024-05-15T12:42:57.775 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.780+0000 7f2fcd4cf640 10 monclient(hunting): _renew_subs 2024-05-15T12:42:57.775 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.780+0000 7f2fcd4cf640 10 monclient(hunting): authenticate will time out at 1341.022666s 2024-05-15T12:42:57.775 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.781+0000 7f2fc6ffd640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2fc80b54b0 0x7f2fc8143c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:57.775 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.781+0000 7f2fc77fe640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2fc81426a0 0x7f2fc8145dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:57.775 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.781+0000 7f2fc6ffd640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2fc80b54b0 0x7f2fc8143c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:53340/0 (socket says 172.21.15.117:53340) 2024-05-15T12:42:57.775 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.781+0000 7f2fc6ffd640 1 -- 172.21.15.117:0/1700230691 learned_addr learned my addr 172.21.15.117:0/1700230691 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:42:57.775 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.781+0000 7f2fc6ffd640 10 monclient(hunting): get_auth_request con 0x7f2fc80b54b0 auth_method 0 2024-05-15T12:42:57.775 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.781+0000 7f2fc6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:57.775 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.781+0000 7f2fc6ffd640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:57.775 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.781+0000 7f2fc6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:57.775 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.782+0000 7f2fc77fe640 10 monclient(hunting): get_auth_request con 0x7f2fc81426a0 auth_method 0 2024-05-15T12:42:57.776 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.782+0000 7f2fc77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:42:57.776 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.782+0000 7f2fc77fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:42:57.776 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.782+0000 7f2fc77fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:42:57.776 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.783+0000 7f2fc77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:57.776 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.783+0000 7f2fc77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:57.776 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.783+0000 7f2fc77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:57.776 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.783+0000 7f2fc6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:42:57.776 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.783+0000 7f2fc6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:42:57.776 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.783+0000 7f2fc6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:42:57.777 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.787+0000 7f2fc6ffd640 10 monclient(hunting): handle_auth_done global_id 4276 payload 824 2024-05-15T12:42:57.777 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.787+0000 7f2fc6ffd640 1 -- 172.21.15.117:0/1700230691 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2fc81426a0 msgr2=0x7f2fc8145dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:57.777 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.787+0000 7f2fc6ffd640 1 --2- 172.21.15.117:0/1700230691 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2fc81426a0 0x7f2fc8145dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:57.777 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.787+0000 7f2fc6ffd640 1 -- 172.21.15.117:0/1700230691 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2fc813f850 msgr2=0x7f2fc8145890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:57.777 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.787+0000 7f2fc6ffd640 1 --2- 172.21.15.117:0/1700230691 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2fc813f850 0x7f2fc8145890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:57.777 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.787+0000 7f2fc6ffd640 10 monclient: _finish_hunting 0 2024-05-15T12:42:57.777 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.787+0000 7f2fc6ffd640 1 monclient: found mon.noname-b 2024-05-15T12:42:57.777 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.787+0000 7f2fc6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:42:57.777 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.787+0000 7f2fc6ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2024-05-15T12:42:57.777 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.787+0000 7f2fc6ffd640 1 -- 172.21.15.117:0/1700230691 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc80b8dd0 con 0x7f2fc80b54b0 2024-05-15T12:42:57.783 INFO:tasks.ceph.osd.4.smithi178.stderr:2024-05-15T12:42:57.790+0000 ac84640 -1 osd.4 0 waiting for initial osdmap 2024-05-15T12:42:57.792 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.802+0000 7f2fc6ffd640 1 --2- 172.21.15.117:0/1700230691 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2fc80b54b0 0x7f2fc8143c50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2fb8034040 tx=0x7f2fb80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:42:57.792 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.802+0000 7f2faf7fe640 10 client.?.objecter ms_handle_connect 0x7f2fc80b54b0 2024-05-15T12:42:57.792 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.802+0000 7f2faf7fe640 10 client.?.objecter resend_mon_ops 2024-05-15T12:42:57.801 INFO:tasks.ceph.osd.1.smithi117.stderr:2024-05-15T12:42:57.618+0000 5dce500 -1 bdev(0x18e82d10 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 2 2024-05-15T12:42:57.802 INFO:tasks.ceph.osd.1.smithi117.stderr:2024-05-15T12:42:57.770+0000 5dce500 -1 bdev(0x18e82d10 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-05-15T12:42:57.807 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.816+0000 7f2faf7fe640 1 -- 172.21.15.117:0/1700230691 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb8032600 con 0x7f2fc80b54b0 2024-05-15T12:42:57.807 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.816+0000 7f2faf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:57.807 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.816+0000 7f2faf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:42:57.807 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.816+0000 7f2faf7fe640 10 monclient: dump: 2024-05-15T12:42:57.807 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-05-15T12:42:57.807 INFO:teuthology.misc.health.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:57.807 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:57.807 INFO:teuthology.misc.health.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:57.807 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:57.808 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-05-15T12:42:57.808 INFO:teuthology.misc.health.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:57.808 INFO:teuthology.misc.health.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:57.808 INFO:teuthology.misc.health.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:57.808 INFO:teuthology.misc.health.smithi117.stderr: 2024-05-15T12:42:57.808 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.816+0000 7f2faf7fe640 10 monclient: _finish_auth 0 2024-05-15T12:42:57.808 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.816+0000 7f2faf7fe640 10 monclient: _check_auth_tickets 2024-05-15T12:42:57.808 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.816+0000 7f2faf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:42:57.808 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.816+0000 7f2faf7fe640 1 -- 172.21.15.117:0/1700230691 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fb80327c0 con 0x7f2fc80b54b0 2024-05-15T12:42:57.808 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.816+0000 7f2faf7fe640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:42:57.808 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.816+0000 7f2faf7fe640 1 -- 172.21.15.117:0/1700230691 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb8032bb0 con 0x7f2fc80b54b0 2024-05-15T12:42:57.808 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.816+0000 7f2faf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:42:57.808 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.816+0000 7f2faf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:42:57.808 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.816+0000 7f2faf7fe640 10 monclient: dump: 2024-05-15T12:42:57.808 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-05-15T12:42:57.809 INFO:teuthology.misc.health.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:42:57.809 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:57.809 INFO:teuthology.misc.health.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:42:57.809 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:42:57.809 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-05-15T12:42:57.809 INFO:teuthology.misc.health.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:42:57.809 INFO:teuthology.misc.health.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:42:57.809 INFO:teuthology.misc.health.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:42:57.809 INFO:teuthology.misc.health.smithi117.stderr: 2024-05-15T12:42:57.809 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.816+0000 7f2fcd4cf640 5 monclient: authenticate success, global_id 4276 2024-05-15T12:42:57.809 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.816+0000 7f2fcd4cf640 10 monclient: _renew_subs 2024-05-15T12:42:57.809 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.816+0000 7f2fcd4cf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:42:57.809 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.816+0000 7f2fc57fa640 4 set_mon_vals no callback set 2024-05-15T12:42:57.809 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.816+0000 7f2fcd4cf640 1 -- 172.21.15.117:0/1700230691 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fc8144dd0 con 0x7f2fc80b54b0 2024-05-15T12:42:57.809 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.816+0000 7f2fcd4cf640 10 client.4276.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:42:57.810 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.816+0000 7f2fcd4cf640 10 monclient: _renew_subs 2024-05-15T12:42:57.810 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.816+0000 7f2fcd4cf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:42:57.810 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.816+0000 7f2fcd4cf640 1 -- 172.21.15.117:0/1700230691 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fc8146310 con 0x7f2fc80b54b0 2024-05-15T12:42:57.810 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.816+0000 7f2fcd4cf640 1 librados: init done 2024-05-15T12:42:57.810 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.817+0000 7f2fcd4cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:42:57.811 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.817+0000 7f2fcd4cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:42:57.811 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.817+0000 7f2fcd4cf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:42:57.811 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:57.817+0000 7f2fcd4cf640 1 -- 172.21.15.117:0/1700230691 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fc81464e0 con 0x7f2fc80b54b0 2024-05-15T12:42:57.811 INFO:tasks.ceph.osd.0.smithi117.stderr:2024-05-15T12:42:57.731+0000 5dce500 -1 bdev(0x18e83880 /var/lib/ceph/osd/ceph-0/block) aio_submit retries 1 2024-05-15T12:42:57.831 INFO:tasks.ceph.osd.1.smithi117.stderr:2024-05-15T12:42:57.812+0000 5dce500 -1 bdev(0x18e82d10 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 2 2024-05-15T12:42:57.896 INFO:tasks.ceph.osd.1.smithi117.stderr:2024-05-15T12:42:57.841+0000 5dce500 -1 bdev(0x18e82d10 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-05-15T12:42:57.897 INFO:tasks.ceph.osd.0.smithi117.stderr:2024-05-15T12:42:57.743+0000 5dce500 -1 bdev(0x18e83880 /var/lib/ceph/osd/ceph-0/block) aio_submit retries 1 2024-05-15T12:42:57.907 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-05-15T12:42:57.511+0000 5dce500 -1 bdev(0x18e82d10 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-05-15T12:42:57.908 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-05-15T12:42:57.917+0000 5dce500 -1 bdev(0x18e82d10 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-05-15T12:42:57.963 INFO:tasks.ceph.osd.1.smithi117.stderr:2024-05-15T12:42:57.909+0000 5dce500 -1 bdev(0x18e82d10 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-05-15T12:42:58.028 INFO:tasks.ceph.osd.1.smithi117.stderr:2024-05-15T12:42:57.973+0000 5dce500 -1 bdev(0x18e82d10 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-05-15T12:42:58.064 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:58.073+0000 7f2faf7fe640 1 -- 172.21.15.117:0/1700230691 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 4) v1 ==== 89722+0+0 (secure 0 0 0) 0x7f2fb804c060 con 0x7f2fc80b54b0 2024-05-15T12:42:58.064 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:58.074+0000 7f2faf7fe640 1 --2- 172.21.15.117:0/1700230691 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f2fb406ce20 0x7f2fb406f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:42:58.064 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:58.074+0000 7f2fc67fc640 1 --2- 172.21.15.117:0/1700230691 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f2fb406ce20 0x7f2fb406f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:42:58.064 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:58.074+0000 7f2fc67fc640 10 monclient: get_auth_request con 0x7f2fb406ce20 auth_method 0 2024-05-15T12:42:58.065 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:58.075+0000 7f2fc67fc640 1 --2- 172.21.15.117:0/1700230691 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f2fb406ce20 0x7f2fb406f2e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc000f60 tx=0x7f2fbc0078f0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:42:58.065 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:58.075+0000 7f2faf7fe640 10 client.4276.objecter ms_handle_connect 0x7f2fb406ce20 2024-05-15T12:42:58.068 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:58.078+0000 7f2faf7fe640 1 -- 172.21.15.117:0/1700230691 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3457+0+0 (secure 0 0 0) 0x7f2fb8078040 con 0x7f2fc80b54b0 2024-05-15T12:42:58.068 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:58.078+0000 7f2faf7fe640 10 client.4276.objecter ms_dispatch 0x7f2fc8000b90 osd_map(18..18 src has 1..18) v4 2024-05-15T12:42:58.068 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:58.078+0000 7f2faf7fe640 3 client.4276.objecter handle_osd_map got epochs [18,18] > 0 2024-05-15T12:42:58.069 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:58.078+0000 7f2faf7fe640 3 client.4276.objecter handle_osd_map decoding full epoch 18 2024-05-15T12:42:58.069 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:58.078+0000 7f2faf7fe640 20 client.4276.objecter dump_active .. 0 homeless 2024-05-15T12:42:58.111 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-05-15T12:42:58.104+0000 5dce500 -1 bdev(0x18e82d10 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-05-15T12:42:58.132 INFO:tasks.ceph.osd.1.smithi117.stderr:2024-05-15T12:42:58.038+0000 5dce500 -1 bdev(0x18e82d10 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-05-15T12:42:58.173 INFO:tasks.ceph.osd.6.smithi178.stderr:2024-05-15T12:42:58.169+0000 14268640 -1 osd.6 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-15T12:42:58.180 INFO:tasks.ceph.osd.0.smithi117.stderr:2024-05-15T12:42:57.821+0000 5dce500 -1 bdev(0x18e83880 /var/lib/ceph/osd/ceph-0/block) aio_submit retries 1 2024-05-15T12:42:58.183 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-05-15T12:42:58.129+0000 5dce500 -1 bdev(0x18e82d10 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 2 2024-05-15T12:42:58.187 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:58.197+0000 7f2faf7fe640 1 -- 172.21.15.117:0/1700230691 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f2fb8040070 con 0x7f2fc80b54b0 2024-05-15T12:42:58.187 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:58.197+0000 7f2faf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:42:58.187 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:58.197+0000 7f2faf7fe640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:42:58.217 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-05-15T12:42:58.199+0000 5dce500 -1 bdev(0x18e82d10 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 2 2024-05-15T12:42:58.233 INFO:tasks.ceph.osd.4.smithi178.stderr:2024-05-15T12:42:58.223+0000 14268640 -1 osd.4 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-15T12:42:58.247 INFO:tasks.ceph.osd.0.smithi117.stderr:2024-05-15T12:42:57.910+0000 5dce500 -1 bdev(0x18e83880 /var/lib/ceph/osd/ceph-0/block) aio_submit retries 1 2024-05-15T12:42:58.248 INFO:tasks.ceph.osd.0.smithi117.stderr:2024-05-15T12:42:58.191+0000 5dce500 -1 bdev(0x18e83880 /var/lib/ceph/osd/ceph-0/block) aio_submit retries 1 2024-05-15T12:42:58.274 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-05-15T12:42:58.227+0000 5dce500 -1 bdev(0x18e82d10 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-05-15T12:42:58.336 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-05-15T12:42:58.249+0000 5dce500 -1 bdev(0x18e82d10 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-05-15T12:42:58.336 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-05-15T12:42:58.267+0000 5dce500 -1 bdev(0x18e82d10 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-05-15T12:42:58.337 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-05-15T12:42:58.286+0000 5dce500 -1 bdev(0x18e82d10 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-05-15T12:42:58.426 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:58.436+0000 7f2fcd4cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:42:58.426 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:58.436+0000 7f2fcd4cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:42:58.426 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:58.436+0000 7f2fcd4cf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:42:58.426 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:58.436+0000 7f2fcd4cf640 1 -- 172.21.15.117:0/1700230691 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2fc80b8b30 con 0x7f2fc80b54b0 2024-05-15T12:42:58.515 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:58.525+0000 7f2faf7fe640 1 -- 172.21.15.117:0/1700230691 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+10446 (secure 0 0 0) 0x7f2fb807a6a0 con 0x7f2fc80b54b0 2024-05-15T12:42:58.515 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:58.525+0000 7f2faf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:42:58.515 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:58.525+0000 7f2faf7fe640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:42:58.515 INFO:teuthology.misc.health.smithi117.stdout: 2024-05-15T12:42:58.516 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":18,"fsid":"8817f6cf-d933-4825-b972-575ef59f5ba9","created":"2024-05-15T12:42:04.264009+0000","modified":"2024-05-15T12:42:55.630069+0000","last_up_change":"2024-05-15T12:42:51.588869+0000","last_in_change":"2024-05-15T12:42:09.879357+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-15T12:42:52.250376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.690000057220459,"score_stable":7.690000057220459,"optimal_score":0.12999999523162842,"raw_score_acting":1,"raw_score_stable":1,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aaffd6d9-1a00-4793-9d1f-65a30501bec4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"efd4e8ed-699c-4ccb-b989-cfadc7aaed35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"708e4e3e-7e79-480f-9dbf-7b972c2f2811","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e7b5d7d7-515c-42d9-9c41-b1ca3977865e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3842021926},{"type":"v1","addr":"172.21.15.117:6803","nonce":3842021926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3842021926},{"type":"v1","addr":"172.21.15.117:6807","nonce":3842021926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":3842021926},{"type":"v1","addr":"172.21.15.117:6814","nonce":3842021926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":3842021926},{"type":"v1","addr":"172.21.15.117:6809","nonce":3842021926}]},"public_addr":"172.21.15.117:6803/3842021926","cluster_addr":"172.21.15.117:6807/3842021926","heartbeat_back_addr":"172.21.15.117:6814/3842021926","heartbeat_front_addr":"172.21.15.117:6809/3842021926","state":["exists","up"]},{"osd":4,"uuid":"fa647c2c-942b-4be3-afb9-6dba6b0a8cf3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5f9bc37-4bd6-4643-8f0d-e53c5f2c8782","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":3713764498},{"type":"v1","addr":"172.21.15.178:6817","nonce":3713764498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":3713764498},{"type":"v1","addr":"172.21.15.178:6819","nonce":3713764498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":3713764498},{"type":"v1","addr":"172.21.15.178:6823","nonce":3713764498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":3713764498},{"type":"v1","addr":"172.21.15.178:6821","nonce":3713764498}]},"public_addr":"172.21.15.178:6817/3713764498","cluster_addr":"172.21.15.178:6819/3713764498","heartbeat_back_addr":"172.21.15.178:6823/3713764498","heartbeat_front_addr":"172.21.15.178:6821/3713764498","state":["exists","up"]},{"osd":6,"uuid":"d5af075d-3eaa-4cb5-b6e9-6f01c201f2ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"323233c7-a3af-4fb0-ac70-de523a279b44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":3049776526},{"type":"v1","addr":"172.21.15.178:6809","nonce":3049776526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":3049776526},{"type":"v1","addr":"172.21.15.178:6811","nonce":3049776526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":3049776526},{"type":"v1","addr":"172.21.15.178:6815","nonce":3049776526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":3049776526},{"type":"v1","addr":"172.21.15.178:6813","nonce":3049776526}]},"public_addr":"172.21.15.178:6809/3049776526","cluster_addr":"172.21.15.178:6811/3049776526","heartbeat_back_addr":"172.21.15.178:6815/3049776526","heartbeat_front_addr":"172.21.15.178:6813/3049776526","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:42:48.243508+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:42:47.225543+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:42:47.675658+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-15T12:42:58.519 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:58.527+0000 7f2fcd4cf640 10 librados: watch_flush enter 2024-05-15T12:42:58.519 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:58.527+0000 7f2fcd4cf640 10 librados: watch_flush exit 2024-05-15T12:42:58.519 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:58.527+0000 7f2fcd4cf640 20 client.4276.objecter shutdown clearing up homeless session... 2024-05-15T12:42:58.519 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:58.527+0000 7f2fcd4cf640 10 client.4276.objecter successfully canceled tick 2024-05-15T12:42:58.519 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:58.528+0000 7f2fcd4cf640 1 -- 172.21.15.117:0/1700230691 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f2fb406ce20 msgr2=0x7f2fb406f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:58.519 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:58.528+0000 7f2fcd4cf640 1 --2- 172.21.15.117:0/1700230691 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f2fb406ce20 0x7f2fb406f2e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc000f60 tx=0x7f2fbc0078f0 comp rx=0 tx=0).stop 2024-05-15T12:42:58.519 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:58.528+0000 7f2fcd4cf640 10 monclient: shutdown 2024-05-15T12:42:58.519 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:58.528+0000 7f2fcd4cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:42:58.519 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:58.528+0000 7f2fcd4cf640 1 -- 172.21.15.117:0/1700230691 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2fc80b54b0 msgr2=0x7f2fc8143c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:42:58.519 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:58.528+0000 7f2fcd4cf640 1 --2- 172.21.15.117:0/1700230691 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2fc80b54b0 0x7f2fc8143c50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2fb8034040 tx=0x7f2fb80046c0 comp rx=0 tx=0).stop 2024-05-15T12:42:58.519 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:58.528+0000 7f2fcd4cf640 1 -- 172.21.15.117:0/1700230691 shutdown_connections 2024-05-15T12:42:58.519 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:58.528+0000 7f2fcd4cf640 1 --2- 172.21.15.117:0/1700230691 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2fc81426a0 0x7f2fc8145dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:58.520 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:58.528+0000 7f2fcd4cf640 1 --2- 172.21.15.117:0/1700230691 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2fc813f850 0x7f2fc8145890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:58.520 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:58.528+0000 7f2fcd4cf640 1 --2- 172.21.15.117:0/1700230691 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f2fb406ce20 0x7f2fb406f2e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:58.520 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:58.528+0000 7f2fcd4cf640 1 --2- 172.21.15.117:0/1700230691 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2fc80b54b0 0x7f2fc8143c50 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:42:58.520 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:58.528+0000 7f2fcd4cf640 1 -- 172.21.15.117:0/1700230691 >> 172.21.15.117:0/1700230691 conn(0x7f2fc806d370 msgr2=0x7f2fc813fc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:42:58.520 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:58.528+0000 7f2fcd4cf640 1 -- 172.21.15.117:0/1700230691 shutdown_connections 2024-05-15T12:42:58.520 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:58.528+0000 7f2fcd4cf640 1 -- 172.21.15.117:0/1700230691 wait complete. 2024-05-15T12:42:58.520 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:42:58.528+0000 7f2fcd4cf640 1 librados: shutdown 2024-05-15T12:42:58.530 DEBUG:teuthology.misc:3 of 8 OSDs are up 2024-05-15T12:42:59.187 INFO:tasks.ceph.osd.1.smithi117.stderr:2024-05-15T12:42:59.190+0000 ac84640 -1 osd.1 0 waiting for initial osdmap 2024-05-15T12:42:59.431 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-05-15T12:42:59.437+0000 ac84640 -1 osd.2 0 waiting for initial osdmap 2024-05-15T12:42:59.466 INFO:tasks.ceph.osd.0.smithi117.stderr:2024-05-15T12:42:59.470+0000 c49a640 -1 osd.0 0 waiting for initial osdmap 2024-05-15T12:42:59.673 INFO:tasks.ceph.osd.1.smithi117.stderr:2024-05-15T12:42:59.669+0000 14268640 -1 osd.1 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-15T12:42:59.949 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-05-15T12:42:59.945+0000 14268640 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-15T12:43:00.104 INFO:tasks.ceph.osd.0.smithi117.stderr:2024-05-15T12:43:00.095+0000 14268640 -1 osd.0 19 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-15T12:43:02.018 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:01:09.196 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:43:02.119 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:01:09.290 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:43:02.148 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:01:09.327 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:43:02.176 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:01:09.348 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:43:02.212 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:01:09.390 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:43:02.279 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:01:09.458 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:43:02.417 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:01:09.589 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:43:02.436 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:01:09.615 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:43:02.569 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:01:09.747 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:43:05.336 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-15T12:43:05.428 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.437+0000 7f4adc587640 10 monclient: get_monmap_and_config 2024-05-15T12:43:05.428 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.437+0000 7f4adc587640 10 monclient: build_initial_monmap 2024-05-15T12:43:05.428 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.437+0000 7f4adc587640 10 monclient: monmap: 2024-05-15T12:43:05.428 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-05-15T12:43:05.428 INFO:teuthology.misc.health.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:05.428 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-05-15T12:43:05.438801+0000 2024-05-15T12:43:05.428 INFO:teuthology.misc.health.smithi117.stderr:created 2024-05-15T12:43:05.438801+0000 2024-05-15T12:43:05.429 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:05.429 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:05.429 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T12:43:05.429 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T12:43:05.429 INFO:teuthology.misc.health.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T12:43:05.429 INFO:teuthology.misc.health.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T12:43:05.429 INFO:teuthology.misc.health.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T12:43:05.429 INFO:teuthology.misc.health.smithi117.stderr: 2024-05-15T12:43:05.429 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.438+0000 7f4adc587640 1 Processor -- start 2024-05-15T12:43:05.429 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.438+0000 7f4adc587640 1 -- start start 2024-05-15T12:43:05.429 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.438+0000 7f4adc587640 10 monclient: init 2024-05-15T12:43:05.430 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.439+0000 7f4adc587640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:05.430 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.439+0000 7f4adc587640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-05-15T12:43:05.430 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.439+0000 7f4adc587640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4ad406af80 addr v1:172.21.15.117:6789/0 2024-05-15T12:43:05.431 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.439+0000 7f4adc587640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4ad40b0680 addr v1:172.21.15.178:6789/0 2024-05-15T12:43:05.431 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.439+0000 7f4adc587640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f4ad413ffe0 0x7f4ad41463d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:05.431 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.439+0000 7f4adc587640 10 monclient(hunting): picked mon.noname-a con 0x7f4ad413ffe0 addr v2:172.21.15.117:3300/0 2024-05-15T12:43:05.431 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.439+0000 7f4adc587640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ad40b1430 con 0x7f4ad406af80 2024-05-15T12:43:05.431 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.439+0000 7f4adc587640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ad4072fc0 con 0x7f4ad40b0680 2024-05-15T12:43:05.431 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.439+0000 7f4adc587640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:05.431 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.439+0000 7f4adc587640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ad406b360 con 0x7f4ad413ffe0 2024-05-15T12:43:05.431 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.439+0000 7f4adc587640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:05.431 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.439+0000 7f4adc587640 10 monclient(hunting): authenticate will time out at 1348.680966s 2024-05-15T12:43:05.431 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.439+0000 7f4adab24640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f4ad413ffe0 0x7f4ad41463d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:05.431 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.439+0000 7f4ad9b22640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f4ad40b0680 0x7f4ad4145cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.117:49162/0 (socket says 172.21.15.117:49162) 2024-05-15T12:43:05.431 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.439+0000 7f4ad9b22640 1 -- 172.21.15.117:0/2777024545 learned_addr learned my addr 172.21.15.117:0/2777024545 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-05-15T12:43:05.431 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.440+0000 7f4adab24640 10 monclient(hunting): get_auth_request con 0x7f4ad413ffe0 auth_method 0 2024-05-15T12:43:05.431 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.440+0000 7f4adab24640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:05.432 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.440+0000 7f4adab24640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:05.432 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.440+0000 7f4adab24640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:05.432 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.441+0000 7f4adab24640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:05.432 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.441+0000 7f4adab24640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:05.432 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.441+0000 7f4adab24640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:05.433 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.442+0000 7f4ad9321640 1 -- 172.21.15.117:0/2777024545 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 77209892 0 0) 0x7f4ad4072fc0 con 0x7f4ad40b0680 2024-05-15T12:43:05.433 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.442+0000 7f4ad9321640 10 monclient(hunting): _negotiate 2024-05-15T12:43:05.433 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.442+0000 7f4ad9321640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:05.433 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.442+0000 7f4ad9321640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:05.433 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.442+0000 7f4ad9321640 10 monclient(hunting): my global_id is 4282 2024-05-15T12:43:05.433 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.442+0000 7f4ad9321640 1 -- 172.21.15.117:0/2777024545 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ac0003520 con 0x7f4ad40b0680 2024-05-15T12:43:05.434 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.444+0000 7f4ad9321640 1 -- 172.21.15.117:0/2777024545 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3602205348 0 0) 0x7f4ad40b1430 con 0x7f4ad406af80 2024-05-15T12:43:05.434 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.444+0000 7f4ad9321640 10 monclient(hunting): _negotiate 2024-05-15T12:43:05.434 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.444+0000 7f4ad9321640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:05.434 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.444+0000 7f4ad9321640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:05.434 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.444+0000 7f4ad9321640 10 monclient(hunting): my global_id is 4251 2024-05-15T12:43:05.434 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.444+0000 7f4ad9321640 1 -- 172.21.15.117:0/2777024545 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ad4072fc0 con 0x7f4ad406af80 2024-05-15T12:43:05.436 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.445+0000 7f4adab24640 10 monclient(hunting): handle_auth_done global_id 4248 payload 274 2024-05-15T12:43:05.436 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.446+0000 7f4adab24640 1 -- 172.21.15.117:0/2777024545 >> v1:172.21.15.178:6789/0 conn(0x7f4ad40b0680 legacy=0x7f4ad4145cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:05.436 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.446+0000 7f4adab24640 1 -- 172.21.15.117:0/2777024545 >> v1:172.21.15.117:6789/0 conn(0x7f4ad406af80 legacy=0x7f4ad413f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:05.436 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.446+0000 7f4adab24640 10 monclient: _finish_hunting 0 2024-05-15T12:43:05.436 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.446+0000 7f4adab24640 1 monclient: found mon.noname-a 2024-05-15T12:43:05.436 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.446+0000 7f4adab24640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:05.436 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.446+0000 7f4adab24640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.117:3300/0 2024-05-15T12:43:05.436 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.446+0000 7f4adab24640 1 -- 172.21.15.117:0/2777024545 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ad40affe0 con 0x7f4ad413ffe0 2024-05-15T12:43:05.440 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.450+0000 7f4adab24640 1 --2- 172.21.15.117:0/2777024545 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4ad413ffe0 0x7f4ad41463d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4acc009dc0 tx=0x7f4acc02bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cc1b425cf2eda51 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:05.446 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.455+0000 7f4ad9321640 1 -- 172.21.15.117:0/2777024545 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4acc039070 con 0x7f4ad413ffe0 2024-05-15T12:43:05.446 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.455+0000 7f4ad9321640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:05.446 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.455+0000 7f4ad9321640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T12:43:05.446 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.455+0000 7f4ad9321640 10 monclient: dump: 2024-05-15T12:43:05.446 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-05-15T12:43:05.446 INFO:teuthology.misc.health.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:05.446 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:05.446 INFO:teuthology.misc.health.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:05.446 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:05.446 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:05.446 INFO:teuthology.misc.health.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:05.447 INFO:teuthology.misc.health.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:05.447 INFO:teuthology.misc.health.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:05.447 INFO:teuthology.misc.health.smithi117.stderr: 2024-05-15T12:43:05.447 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.455+0000 7f4ad9321640 10 monclient: _finish_auth 0 2024-05-15T12:43:05.447 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.455+0000 7f4ad9321640 10 monclient: _check_auth_tickets 2024-05-15T12:43:05.447 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.455+0000 7f4ad9321640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:05.447 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.455+0000 7f4ad9321640 1 -- 172.21.15.117:0/2777024545 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4acc02ccd0 con 0x7f4ad413ffe0 2024-05-15T12:43:05.447 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.455+0000 7f4adc587640 5 monclient: authenticate success, global_id 4248 2024-05-15T12:43:05.447 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.455+0000 7f4adc587640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:43:05.447 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.455+0000 7f4ad9321640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:05.447 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.455+0000 7f4adc587640 10 monclient: get_monmap_and_config success 2024-05-15T12:43:05.447 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.455+0000 7f4adc587640 4 set_mon_vals no callback set 2024-05-15T12:43:05.448 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.455+0000 7f4ad9321640 1 -- 172.21.15.117:0/2777024545 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4acc034660 con 0x7f4ad413ffe0 2024-05-15T12:43:05.448 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.455+0000 7f4ad9321640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:05.448 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.455+0000 7f4ad9321640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T12:43:05.448 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.455+0000 7f4ad9321640 10 monclient: dump: 2024-05-15T12:43:05.448 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-05-15T12:43:05.448 INFO:teuthology.misc.health.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:05.448 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:05.448 INFO:teuthology.misc.health.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:05.448 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:05.448 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:05.449 INFO:teuthology.misc.health.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:05.449 INFO:teuthology.misc.health.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:05.449 INFO:teuthology.misc.health.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:05.449 INFO:teuthology.misc.health.smithi117.stderr: 2024-05-15T12:43:05.449 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.455+0000 7f4adc587640 10 monclient: shutdown 2024-05-15T12:43:05.449 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.455+0000 7f4adc587640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:05.449 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.455+0000 7f4adc587640 1 -- 172.21.15.117:0/2777024545 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4ad413ffe0 msgr2=0x7f4ad41463d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:05.449 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.455+0000 7f4adc587640 1 --2- 172.21.15.117:0/2777024545 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4ad413ffe0 0x7f4ad41463d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4acc009dc0 tx=0x7f4acc02bf20 comp rx=0 tx=0).stop 2024-05-15T12:43:05.449 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.455+0000 7f4adc587640 1 -- 172.21.15.117:0/2777024545 shutdown_connections 2024-05-15T12:43:05.449 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.455+0000 7f4adc587640 1 --2- 172.21.15.117:0/2777024545 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4ad413ffe0 0x7f4ad41463d0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:05.449 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.455+0000 7f4adc587640 1 -- 172.21.15.117:0/2777024545 >> 172.21.15.117:0/2777024545 conn(0x7f4ad40b4fd0 msgr2=0x7f4ad40b53e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:05.450 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.456+0000 7f4adc587640 1 -- 172.21.15.117:0/2777024545 shutdown_connections 2024-05-15T12:43:05.450 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.456+0000 7f4adc587640 1 -- 172.21.15.117:0/2777024545 wait complete. 2024-05-15T12:43:05.454 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.456+0000 7f4adc587640 10 monclient: build_initial_monmap 2024-05-15T12:43:05.454 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.456+0000 7f4adc587640 10 monclient: monmap: 2024-05-15T12:43:05.454 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-05-15T12:43:05.454 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:43:05.454 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-05-15T12:43:05.454 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-05-15T12:43:05.454 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:05.454 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:05.454 INFO:teuthology.misc.health.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T12:43:05.454 INFO:teuthology.misc.health.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T12:43:05.455 INFO:teuthology.misc.health.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T12:43:05.455 INFO:teuthology.misc.health.smithi117.stderr: 2024-05-15T12:43:05.455 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.456+0000 7f4adc587640 1 librados: starting msgr at 2024-05-15T12:43:05.458 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.456+0000 7f4adc587640 1 librados: starting objecter 2024-05-15T12:43:05.458 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.457+0000 7f4adc587640 1 Processor -- start 2024-05-15T12:43:05.458 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.457+0000 7f4adc587640 1 -- start start 2024-05-15T12:43:05.458 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.457+0000 7f4adc587640 1 librados: setting wanted keys 2024-05-15T12:43:05.458 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.457+0000 7f4adc587640 1 librados: calling monclient init 2024-05-15T12:43:05.459 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.457+0000 7f4adc587640 10 monclient: init 2024-05-15T12:43:05.459 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.457+0000 7f4adc587640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:05.459 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.457+0000 7f4adc587640 10 monclient: _add_conns ranks=[0,1,2] 2024-05-15T12:43:05.459 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.457+0000 7f4adc587640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4ad406af80 0x7f4ad413afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:05.459 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.457+0000 7f4adc587640 10 monclient(hunting): picked mon.noname-a con 0x7f4ad406af80 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T12:43:05.459 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.457+0000 7f4adc587640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4ad40b0680 0x7f4ad41db5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:05.459 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.457+0000 7f4adc587640 10 monclient(hunting): picked mon.noname-b con 0x7f4ad40b0680 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:43:05.459 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.457+0000 7f4adc587640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4ad413ffe0 0x7f4ad41dbb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:05.459 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.457+0000 7f4adc587640 10 monclient(hunting): picked mon.noname-c con 0x7f4ad413ffe0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:43:05.459 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.457+0000 7f4adc587640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:05.459 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.457+0000 7f4adc587640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad414e1e0 con 0x7f4ad406af80 2024-05-15T12:43:05.459 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.457+0000 7f4adc587640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:05.460 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.457+0000 7f4adc587640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad414e060 con 0x7f4ad40b0680 2024-05-15T12:43:05.460 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.457+0000 7f4adc587640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:05.460 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.457+0000 7f4adc587640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad414e360 con 0x7f4ad413ffe0 2024-05-15T12:43:05.460 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.457+0000 7f4adc587640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:05.460 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.457+0000 7f4adc587640 10 monclient(hunting): authenticate will time out at 1348.699416s 2024-05-15T12:43:05.460 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.458+0000 7f4adab24640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4ad413ffe0 0x7f4ad41dbb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:05.460 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.458+0000 7f4ad9b22640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4ad40b0680 0x7f4ad41db5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:05.460 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.458+0000 7f4adab24640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4ad413ffe0 0x7f4ad41dbb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.117:48018/0 (socket says 172.21.15.117:48018) 2024-05-15T12:43:05.460 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.458+0000 7f4adab24640 1 -- 172.21.15.117:0/1148063265 learned_addr learned my addr 172.21.15.117:0/1148063265 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:43:05.460 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.458+0000 7f4adab24640 10 monclient(hunting): get_auth_request con 0x7f4ad413ffe0 auth_method 0 2024-05-15T12:43:05.460 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.458+0000 7f4adab24640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:05.460 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.458+0000 7f4adab24640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:05.461 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.458+0000 7f4adab24640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:05.461 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.458+0000 7f4ad9b22640 10 monclient(hunting): get_auth_request con 0x7f4ad40b0680 auth_method 0 2024-05-15T12:43:05.461 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.458+0000 7f4ad9b22640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:05.461 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.458+0000 7f4ad9b22640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:05.461 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.458+0000 7f4ad9b22640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:05.461 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.459+0000 7f4adab24640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:05.461 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.459+0000 7f4adab24640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:05.461 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.459+0000 7f4adab24640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:05.461 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.459+0000 7f4ad9b22640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:05.461 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.459+0000 7f4ad9b22640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:05.461 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.459+0000 7f4ad9b22640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:05.462 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.460+0000 7f4ada323640 1 --2- 172.21.15.117:0/1148063265 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4ad406af80 0x7f4ad413afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:05.462 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.461+0000 7f4ada323640 10 monclient(hunting): get_auth_request con 0x7f4ad406af80 auth_method 0 2024-05-15T12:43:05.462 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.461+0000 7f4ada323640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:05.462 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.461+0000 7f4ada323640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:05.462 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.461+0000 7f4ada323640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:05.462 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.462+0000 7f4ada323640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:05.462 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.462+0000 7f4ada323640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:05.462 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.462+0000 7f4ada323640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:05.462 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.462+0000 7f4adab24640 10 monclient(hunting): handle_auth_done global_id 4235 payload 824 2024-05-15T12:43:05.462 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.462+0000 7f4adab24640 1 -- 172.21.15.117:0/1148063265 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4ad40b0680 msgr2=0x7f4ad41db5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:05.462 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.462+0000 7f4adab24640 1 --2- 172.21.15.117:0/1148063265 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4ad40b0680 0x7f4ad41db5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:05.462 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.462+0000 7f4adab24640 1 -- 172.21.15.117:0/1148063265 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4ad406af80 msgr2=0x7f4ad413afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:05.463 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.462+0000 7f4adab24640 1 --2- 172.21.15.117:0/1148063265 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4ad406af80 0x7f4ad413afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:05.463 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.462+0000 7f4adab24640 10 monclient: _finish_hunting 0 2024-05-15T12:43:05.463 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.462+0000 7f4adab24640 1 monclient: found mon.noname-c 2024-05-15T12:43:05.463 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.462+0000 7f4adab24640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:05.463 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.462+0000 7f4adab24640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.117:3301/0 2024-05-15T12:43:05.463 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.462+0000 7f4adab24640 1 -- 172.21.15.117:0/1148063265 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ad4137ae0 con 0x7f4ad413ffe0 2024-05-15T12:43:05.463 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.462+0000 7f4ad9b22640 1 --2- 172.21.15.117:0/1148063265 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4ad40b0680 0x7f4ad41db5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-15T12:43:05.465 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.466+0000 7f4adab24640 1 --2- 172.21.15.117:0/1148063265 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4ad413ffe0 0x7f4ad41dbb00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f4acc037040 tx=0x7f4acc004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:05.465 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.466+0000 7f4acaffd640 10 client.?.objecter ms_handle_connect 0x7f4ad413ffe0 2024-05-15T12:43:05.465 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.466+0000 7f4acaffd640 10 client.?.objecter resend_mon_ops 2024-05-15T12:43:05.465 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.471+0000 7f4acaffd640 1 -- 172.21.15.117:0/1148063265 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4acc039050 con 0x7f4ad413ffe0 2024-05-15T12:43:05.465 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.471+0000 7f4acaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:05.465 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.471+0000 7f4acaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:43:05.465 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.471+0000 7f4acaffd640 10 monclient: dump: 2024-05-15T12:43:05.465 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-05-15T12:43:05.465 INFO:teuthology.misc.health.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:05.466 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:05.466 INFO:teuthology.misc.health.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:05.466 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:05.466 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:05.466 INFO:teuthology.misc.health.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:05.466 INFO:teuthology.misc.health.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:05.466 INFO:teuthology.misc.health.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:05.466 INFO:teuthology.misc.health.smithi117.stderr: 2024-05-15T12:43:05.466 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.471+0000 7f4acaffd640 10 monclient: _finish_auth 0 2024-05-15T12:43:05.466 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.471+0000 7f4acaffd640 10 monclient: _check_auth_tickets 2024-05-15T12:43:05.466 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.471+0000 7f4acaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:05.466 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.471+0000 7f4acaffd640 1 -- 172.21.15.117:0/1148063265 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4acc045070 con 0x7f4ad413ffe0 2024-05-15T12:43:05.466 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.471+0000 7f4adc587640 5 monclient: authenticate success, global_id 4235 2024-05-15T12:43:05.467 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.471+0000 7f4adc587640 10 monclient: _renew_subs 2024-05-15T12:43:05.467 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.471+0000 7f4adc587640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T12:43:05.467 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.471+0000 7f4adc587640 1 -- 172.21.15.117:0/1148063265 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ad4137dc0 con 0x7f4ad413ffe0 2024-05-15T12:43:05.467 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.471+0000 7f4acaffd640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:05.467 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.471+0000 7f4acaffd640 1 -- 172.21.15.117:0/1148063265 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4acc009450 con 0x7f4ad413ffe0 2024-05-15T12:43:05.467 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.471+0000 7f4acaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:05.467 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.471+0000 7f4acaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T12:43:05.467 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.471+0000 7f4acaffd640 10 monclient: dump: 2024-05-15T12:43:05.467 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-05-15T12:43:05.467 INFO:teuthology.misc.health.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:05.467 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:05.467 INFO:teuthology.misc.health.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:05.467 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:05.467 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:05.467 INFO:teuthology.misc.health.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:05.468 INFO:teuthology.misc.health.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:05.468 INFO:teuthology.misc.health.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:05.468 INFO:teuthology.misc.health.smithi117.stderr: 2024-05-15T12:43:05.468 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.471+0000 7f4ad8b20640 4 set_mon_vals no callback set 2024-05-15T12:43:05.468 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.471+0000 7f4adc587640 10 client.4235.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:43:05.468 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.471+0000 7f4adc587640 10 monclient: _renew_subs 2024-05-15T12:43:05.468 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.471+0000 7f4adc587640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T12:43:05.468 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.471+0000 7f4adc587640 1 -- 172.21.15.117:0/1148063265 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ad4132bc0 con 0x7f4ad413ffe0 2024-05-15T12:43:05.468 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.471+0000 7f4adc587640 1 librados: init done 2024-05-15T12:43:05.468 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.472+0000 7f4adc587640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:05.468 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.472+0000 7f4adc587640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:05.468 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.472+0000 7f4adc587640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T12:43:05.468 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.472+0000 7f4adc587640 1 -- 172.21.15.117:0/1148063265 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ad404f0e0 con 0x7f4ad413ffe0 2024-05-15T12:43:05.715 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.724+0000 7f4acaffd640 1 -- 172.21.15.117:0/1148063265 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f4acc03e850 con 0x7f4ad413ffe0 2024-05-15T12:43:05.715 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.725+0000 7f4acaffd640 1 --2- 172.21.15.117:0/1148063265 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f4ab806ceb0 0x7f4ab806f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:05.715 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.725+0000 7f4ada323640 1 --2- 172.21.15.117:0/1148063265 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f4ab806ceb0 0x7f4ab806f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:05.715 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.725+0000 7f4ada323640 10 monclient: get_auth_request con 0x7f4ab806ceb0 auth_method 0 2024-05-15T12:43:05.715 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.725+0000 7f4acaffd640 10 client.4235.objecter ms_handle_connect 0x7f4ab806ceb0 2024-05-15T12:43:05.715 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.725+0000 7f4ada323640 1 --2- 172.21.15.117:0/1148063265 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f4ab806ceb0 0x7f4ab806f370 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4ad0007fd0 tx=0x7f4ad0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:05.727 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.737+0000 7f4acaffd640 1 -- 172.21.15.117:0/1148063265 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f4acc0822d0 con 0x7f4ad413ffe0 2024-05-15T12:43:05.728 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.737+0000 7f4acaffd640 10 client.4235.objecter ms_dispatch 0x7f4ad4000b90 osd_map(22..22 src has 1..22) v4 2024-05-15T12:43:05.728 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.737+0000 7f4acaffd640 3 client.4235.objecter handle_osd_map got epochs [22,22] > 0 2024-05-15T12:43:05.728 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.737+0000 7f4acaffd640 3 client.4235.objecter handle_osd_map decoding full epoch 22 2024-05-15T12:43:05.728 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.737+0000 7f4acaffd640 20 client.4235.objecter dump_active .. 0 homeless 2024-05-15T12:43:05.728 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.737+0000 7f4acaffd640 1 -- 172.21.15.117:0/1148063265 <== mon.2 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f4acc07bfb0 con 0x7f4ad413ffe0 2024-05-15T12:43:05.728 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.737+0000 7f4acaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:05.728 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.737+0000 7f4acaffd640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:43:05.900 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.910+0000 7f4adc587640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:43:05.900 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.910+0000 7f4adc587640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:43:05.900 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.910+0000 7f4adc587640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T12:43:05.900 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:05.910+0000 7f4adc587640 1 -- 172.21.15.117:0/1148063265 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ad40b3c90 con 0x7f4ad413ffe0 2024-05-15T12:43:05.990 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:06.000+0000 7f4acaffd640 1 -- 172.21.15.117:0/1148063265 <== mon.2 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+12933 (secure 0 0 0) 0x7f4acc080050 con 0x7f4ad413ffe0 2024-05-15T12:43:05.990 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:06.000+0000 7f4acaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:43:05.990 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:06.000+0000 7f4acaffd640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:43:05.990 INFO:teuthology.misc.health.smithi117.stdout: 2024-05-15T12:43:05.991 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":22,"fsid":"8817f6cf-d933-4825-b972-575ef59f5ba9","created":"2024-05-15T12:42:04.264009+0000","modified":"2024-05-15T12:43:03.790596+0000","last_up_change":"2024-05-15T12:43:01.039161+0000","last_in_change":"2024-05-15T12:42:09.879357+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-15T12:42:52.250376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aaffd6d9-1a00-4793-9d1f-65a30501bec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":1673738625},{"type":"v1","addr":"172.21.15.117:6827","nonce":1673738625}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":1673738625},{"type":"v1","addr":"172.21.15.117:6829","nonce":1673738625}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":1673738625},{"type":"v1","addr":"172.21.15.117:6833","nonce":1673738625}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":1673738625},{"type":"v1","addr":"172.21.15.117:6831","nonce":1673738625}]},"public_addr":"172.21.15.117:6827/1673738625","cluster_addr":"172.21.15.117:6829/1673738625","heartbeat_back_addr":"172.21.15.117:6833/1673738625","heartbeat_front_addr":"172.21.15.117:6831/1673738625","state":["exists","up"]},{"osd":1,"uuid":"efd4e8ed-699c-4ccb-b989-cfadc7aaed35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3864975175},{"type":"v1","addr":"172.21.15.117:6819","nonce":3864975175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3864975175},{"type":"v1","addr":"172.21.15.117:6821","nonce":3864975175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3864975175},{"type":"v1","addr":"172.21.15.117:6825","nonce":3864975175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3864975175},{"type":"v1","addr":"172.21.15.117:6823","nonce":3864975175}]},"public_addr":"172.21.15.117:6819/3864975175","cluster_addr":"172.21.15.117:6821/3864975175","heartbeat_back_addr":"172.21.15.117:6825/3864975175","heartbeat_front_addr":"172.21.15.117:6823/3864975175","state":["exists","up"]},{"osd":2,"uuid":"708e4e3e-7e79-480f-9dbf-7b972c2f2811","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1425835241},{"type":"v1","addr":"172.21.15.117:6805","nonce":1425835241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1425835241},{"type":"v1","addr":"172.21.15.117:6811","nonce":1425835241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":1425835241},{"type":"v1","addr":"172.21.15.117:6817","nonce":1425835241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6813","nonce":1425835241},{"type":"v1","addr":"172.21.15.117:6815","nonce":1425835241}]},"public_addr":"172.21.15.117:6805/1425835241","cluster_addr":"172.21.15.117:6811/1425835241","heartbeat_back_addr":"172.21.15.117:6817/1425835241","heartbeat_front_addr":"172.21.15.117:6815/1425835241","state":["exists","up"]},{"osd":3,"uuid":"e7b5d7d7-515c-42d9-9c41-b1ca3977865e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3842021926},{"type":"v1","addr":"172.21.15.117:6803","nonce":3842021926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3842021926},{"type":"v1","addr":"172.21.15.117:6807","nonce":3842021926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":3842021926},{"type":"v1","addr":"172.21.15.117:6814","nonce":3842021926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":3842021926},{"type":"v1","addr":"172.21.15.117:6809","nonce":3842021926}]},"public_addr":"172.21.15.117:6803/3842021926","cluster_addr":"172.21.15.117:6807/3842021926","heartbeat_back_addr":"172.21.15.117:6814/3842021926","heartbeat_front_addr":"172.21.15.117:6809/3842021926","state":["exists","up"]},{"osd":4,"uuid":"fa647c2c-942b-4be3-afb9-6dba6b0a8cf3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":248472851},{"type":"v1","addr":"172.21.15.178:6825","nonce":248472851}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":248472851},{"type":"v1","addr":"172.21.15.178:6827","nonce":248472851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":248472851},{"type":"v1","addr":"172.21.15.178:6831","nonce":248472851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":248472851},{"type":"v1","addr":"172.21.15.178:6829","nonce":248472851}]},"public_addr":"172.21.15.178:6825/248472851","cluster_addr":"172.21.15.178:6827/248472851","heartbeat_back_addr":"172.21.15.178:6831/248472851","heartbeat_front_addr":"172.21.15.178:6829/248472851","state":["exists","up"]},{"osd":5,"uuid":"c5f9bc37-4bd6-4643-8f0d-e53c5f2c8782","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":3713764498},{"type":"v1","addr":"172.21.15.178:6817","nonce":3713764498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":3713764498},{"type":"v1","addr":"172.21.15.178:6819","nonce":3713764498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":3713764498},{"type":"v1","addr":"172.21.15.178:6823","nonce":3713764498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":3713764498},{"type":"v1","addr":"172.21.15.178:6821","nonce":3713764498}]},"public_addr":"172.21.15.178:6817/3713764498","cluster_addr":"172.21.15.178:6819/3713764498","heartbeat_back_addr":"172.21.15.178:6823/3713764498","heartbeat_front_addr":"172.21.15.178:6821/3713764498","state":["exists","up"]},{"osd":6,"uuid":"d5af075d-3eaa-4cb5-b6e9-6f01c201f2ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1383036222},{"type":"v1","addr":"172.21.15.178:6801","nonce":1383036222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1383036222},{"type":"v1","addr":"172.21.15.178:6803","nonce":1383036222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1383036222},{"type":"v1","addr":"172.21.15.178:6807","nonce":1383036222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1383036222},{"type":"v1","addr":"172.21.15.178:6805","nonce":1383036222}]},"public_addr":"172.21.15.178:6801/1383036222","cluster_addr":"172.21.15.178:6803/1383036222","heartbeat_back_addr":"172.21.15.178:6807/1383036222","heartbeat_front_addr":"172.21.15.178:6805/1383036222","state":["exists","up"]},{"osd":7,"uuid":"323233c7-a3af-4fb0-ac70-de523a279b44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":3049776526},{"type":"v1","addr":"172.21.15.178:6809","nonce":3049776526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":3049776526},{"type":"v1","addr":"172.21.15.178:6811","nonce":3049776526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":3049776526},{"type":"v1","addr":"172.21.15.178:6815","nonce":3049776526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":3049776526},{"type":"v1","addr":"172.21.15.178:6813","nonce":3049776526}]},"public_addr":"172.21.15.178:6809/3049776526","cluster_addr":"172.21.15.178:6811/3049776526","heartbeat_back_addr":"172.21.15.178:6815/3049776526","heartbeat_front_addr":"172.21.15.178:6813/3049776526","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:42:48.855951+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:42:49.166018+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:42:48.891180+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:42:48.243508+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:42:47.750264+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:42:47.225543+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:42:47.826018+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:42:47.675658+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-15T12:43:05.993 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:06.002+0000 7f4adc587640 10 librados: watch_flush enter 2024-05-15T12:43:05.993 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:06.002+0000 7f4adc587640 10 librados: watch_flush exit 2024-05-15T12:43:05.993 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:06.002+0000 7f4adc587640 20 client.4235.objecter shutdown clearing up homeless session... 2024-05-15T12:43:05.993 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:06.002+0000 7f4adc587640 10 client.4235.objecter successfully canceled tick 2024-05-15T12:43:05.993 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:06.002+0000 7f4adc587640 1 -- 172.21.15.117:0/1148063265 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f4ab806ceb0 msgr2=0x7f4ab806f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:05.993 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:06.002+0000 7f4adc587640 1 --2- 172.21.15.117:0/1148063265 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f4ab806ceb0 0x7f4ab806f370 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4ad0007fd0 tx=0x7f4ad0009040 comp rx=0 tx=0).stop 2024-05-15T12:43:05.993 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:06.002+0000 7f4adc587640 10 monclient: shutdown 2024-05-15T12:43:05.993 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:06.002+0000 7f4adc587640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:05.993 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:06.002+0000 7f4adc587640 1 -- 172.21.15.117:0/1148063265 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4ad413ffe0 msgr2=0x7f4ad41dbb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:05.994 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:06.002+0000 7f4adc587640 1 --2- 172.21.15.117:0/1148063265 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4ad413ffe0 0x7f4ad41dbb00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f4acc037040 tx=0x7f4acc004a40 comp rx=0 tx=0).stop 2024-05-15T12:43:05.994 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:06.002+0000 7f4adc587640 1 -- 172.21.15.117:0/1148063265 shutdown_connections 2024-05-15T12:43:05.994 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:06.002+0000 7f4adc587640 1 --2- 172.21.15.117:0/1148063265 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4ad413ffe0 0x7f4ad41dbb00 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:05.994 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:06.002+0000 7f4adc587640 1 --2- 172.21.15.117:0/1148063265 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f4ab806ceb0 0x7f4ab806f370 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:05.994 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:06.002+0000 7f4adc587640 1 --2- 172.21.15.117:0/1148063265 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4ad40b0680 0x7f4ad41db5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:05.994 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:06.002+0000 7f4adc587640 1 --2- 172.21.15.117:0/1148063265 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4ad406af80 0x7f4ad413afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:05.994 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:06.002+0000 7f4adc587640 1 -- 172.21.15.117:0/1148063265 >> 172.21.15.117:0/1148063265 conn(0x7f4ad40b4fd0 msgr2=0x7f4ad40b3400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:05.994 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:06.002+0000 7f4adc587640 1 -- 172.21.15.117:0/1148063265 shutdown_connections 2024-05-15T12:43:05.994 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:06.002+0000 7f4adc587640 1 -- 172.21.15.117:0/1148063265 wait complete. 2024-05-15T12:43:05.994 INFO:teuthology.misc.health.smithi117.stderr:2024-05-15T12:43:06.002+0000 7f4adc587640 1 librados: shutdown 2024-05-15T12:43:06.004 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-05-15T12:43:06.004 INFO:tasks.ceph:Creating RBD pool 2024-05-15T12:43:06.004 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph osd pool create rbd 8 2024-05-15T12:43:06.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.117+0000 7f8a19f28640 10 monclient: get_monmap_and_config 2024-05-15T12:43:06.108 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.117+0000 7f8a19f28640 10 monclient: build_initial_monmap 2024-05-15T12:43:06.108 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.117+0000 7f8a19f28640 10 monclient: monmap: 2024-05-15T12:43:06.108 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:06.108 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:06.108 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:43:06.118967+0000 2024-05-15T12:43:06.108 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:43:06.118967+0000 2024-05-15T12:43:06.108 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:06.108 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:06.108 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T12:43:06.108 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T12:43:06.108 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T12:43:06.108 INFO:teuthology.orchestra.run.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T12:43:06.109 INFO:teuthology.orchestra.run.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T12:43:06.109 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:06.110 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.118+0000 7f8a19f28640 1 Processor -- start 2024-05-15T12:43:06.110 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.118+0000 7f8a19f28640 1 -- start start 2024-05-15T12:43:06.110 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.118+0000 7f8a19f28640 10 monclient: init 2024-05-15T12:43:06.110 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.119+0000 7f8a19f28640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:06.110 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.119+0000 7f8a19f28640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-05-15T12:43:06.110 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.119+0000 7f8a19f28640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8a140b5180 addr v1:172.21.15.117:6789/0 2024-05-15T12:43:06.111 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.119+0000 7f8a19f28640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8a1413ff40 addr v1:172.21.15.178:6789/0 2024-05-15T12:43:06.111 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.119+0000 7f8a19f28640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8a14142d90 0x7f8a140a9980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:06.111 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.119+0000 7f8a19f28640 10 monclient(hunting): picked mon.noname-b con 0x7f8a14142d90 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:43:06.111 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.119+0000 7f8a19f28640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a140aa530 con 0x7f8a140b5180 2024-05-15T12:43:06.111 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.119+0000 7f8a19f28640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a140ab6b0 con 0x7f8a1413ff40 2024-05-15T12:43:06.111 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.119+0000 7f8a19f28640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:06.111 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.119+0000 7f8a19f28640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a140b5560 con 0x7f8a14142d90 2024-05-15T12:43:06.111 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.119+0000 7f8a19f28640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:06.111 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.119+0000 7f8a19f28640 10 monclient(hunting): authenticate will time out at 1349.361377s 2024-05-15T12:43:06.111 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.120+0000 7f8a12ffd640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f8a1413ff40 0x7f8a14142680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.117:49172/0 (socket says 172.21.15.117:49172) 2024-05-15T12:43:06.111 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.120+0000 7f8a12ffd640 1 -- 172.21.15.117:0/1096779389 learned_addr learned my addr 172.21.15.117:0/1096779389 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-05-15T12:43:06.112 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.122+0000 7f8a13fff640 1 --2- 172.21.15.117:0/1096779389 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8a14142d90 0x7f8a140a9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:06.112 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.122+0000 7f8a127fc640 1 -- 172.21.15.117:0/1096779389 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1699639156 0 0) 0x7f8a140aa530 con 0x7f8a140b5180 2024-05-15T12:43:06.112 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.122+0000 7f8a127fc640 10 monclient(hunting): _negotiate 2024-05-15T12:43:06.112 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.122+0000 7f8a127fc640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:06.112 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.122+0000 7f8a127fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:06.112 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.122+0000 7f8a127fc640 10 monclient(hunting): my global_id is 4257 2024-05-15T12:43:06.113 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.122+0000 7f8a127fc640 1 -- 172.21.15.117:0/1096779389 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a00003520 con 0x7f8a140b5180 2024-05-15T12:43:06.113 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.123+0000 7f8a13fff640 10 monclient(hunting): get_auth_request con 0x7f8a14142d90 auth_method 0 2024-05-15T12:43:06.113 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.123+0000 7f8a13fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:06.113 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.123+0000 7f8a13fff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:06.113 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.123+0000 7f8a13fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:06.114 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.124+0000 7f8a13fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:06.114 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.124+0000 7f8a13fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:06.114 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.124+0000 7f8a13fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:06.116 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.126+0000 7f8a13fff640 10 monclient(hunting): handle_auth_done global_id 4238 payload 274 2024-05-15T12:43:06.116 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.126+0000 7f8a13fff640 1 -- 172.21.15.117:0/1096779389 >> v1:172.21.15.178:6789/0 conn(0x7f8a1413ff40 legacy=0x7f8a14142680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:06.116 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.126+0000 7f8a13fff640 1 -- 172.21.15.117:0/1096779389 >> v1:172.21.15.117:6789/0 conn(0x7f8a140b5180 legacy=0x7f8a1413f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:06.116 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.126+0000 7f8a13fff640 10 monclient: _finish_hunting 0 2024-05-15T12:43:06.116 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.126+0000 7f8a13fff640 1 monclient: found mon.noname-b 2024-05-15T12:43:06.116 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.126+0000 7f8a13fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:06.116 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.126+0000 7f8a13fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.117:3301/0 2024-05-15T12:43:06.116 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.126+0000 7f8a13fff640 1 -- 172.21.15.117:0/1096779389 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a140aa010 con 0x7f8a14142d90 2024-05-15T12:43:06.118 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.128+0000 7f8a13fff640 1 --2- 172.21.15.117:0/1096779389 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8a14142d90 0x7f8a140a9980 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8a0c0097d0 tx=0x7f8a0c02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bdf5b882a6752ee9 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:06.122 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.132+0000 7f8a127fc640 1 -- 172.21.15.117:0/1096779389 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a0c035020 con 0x7f8a14142d90 2024-05-15T12:43:06.122 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.132+0000 7f8a127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:06.122 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.132+0000 7f8a127fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:43:06.122 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.132+0000 7f8a127fc640 10 monclient: dump: 2024-05-15T12:43:06.122 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:06.122 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:06.122 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:06.122 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:06.122 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:06.122 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:06.123 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:06.123 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:06.123 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:06.123 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:06.123 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.132+0000 7f8a127fc640 10 monclient: _finish_auth 0 2024-05-15T12:43:06.123 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.132+0000 7f8a127fc640 10 monclient: _check_auth_tickets 2024-05-15T12:43:06.123 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.132+0000 7f8a127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:06.123 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.132+0000 7f8a127fc640 1 -- 172.21.15.117:0/1096779389 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a0c031890 con 0x7f8a14142d90 2024-05-15T12:43:06.123 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.132+0000 7f8a127fc640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:06.123 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.132+0000 7f8a127fc640 1 -- 172.21.15.117:0/1096779389 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a0c031c60 con 0x7f8a14142d90 2024-05-15T12:43:06.123 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.132+0000 7f8a19f28640 5 monclient: authenticate success, global_id 4238 2024-05-15T12:43:06.123 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.132+0000 7f8a19f28640 10 monclient: get_monmap_and_config success 2024-05-15T12:43:06.123 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.132+0000 7f8a19f28640 4 set_mon_vals no callback set 2024-05-15T12:43:06.123 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.132+0000 7f8a127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:06.123 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.132+0000 7f8a127fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T12:43:06.124 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.132+0000 7f8a127fc640 10 monclient: dump: 2024-05-15T12:43:06.124 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:06.124 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:06.124 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:06.124 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:06.124 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:06.124 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:06.124 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:06.125 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:06.125 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:06.125 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:06.128 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.132+0000 7f8a19f28640 10 monclient: shutdown 2024-05-15T12:43:06.128 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.132+0000 7f8a19f28640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:06.128 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.132+0000 7f8a19f28640 1 -- 172.21.15.117:0/1096779389 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8a14142d90 msgr2=0x7f8a140a9980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:06.128 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.132+0000 7f8a19f28640 1 --2- 172.21.15.117:0/1096779389 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8a14142d90 0x7f8a140a9980 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8a0c0097d0 tx=0x7f8a0c02b930 comp rx=0 tx=0).stop 2024-05-15T12:43:06.128 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.133+0000 7f8a19f28640 1 -- 172.21.15.117:0/1096779389 shutdown_connections 2024-05-15T12:43:06.128 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.133+0000 7f8a19f28640 1 --2- 172.21.15.117:0/1096779389 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8a14142d90 0x7f8a140a9980 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:06.129 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.133+0000 7f8a19f28640 1 -- 172.21.15.117:0/1096779389 >> 172.21.15.117:0/1096779389 conn(0x7f8a14066ed0 msgr2=0x7f8a140672e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:06.129 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.133+0000 7f8a19f28640 1 -- 172.21.15.117:0/1096779389 shutdown_connections 2024-05-15T12:43:06.129 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.133+0000 7f8a19f28640 1 -- 172.21.15.117:0/1096779389 wait complete. 2024-05-15T12:43:06.129 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.133+0000 7f8a19f28640 10 monclient: build_initial_monmap 2024-05-15T12:43:06.129 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.133+0000 7f8a19f28640 10 monclient: monmap: 2024-05-15T12:43:06.129 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:06.129 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:43:06.129 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-05-15T12:43:06.129 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-05-15T12:43:06.129 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:06.129 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:06.129 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T12:43:06.129 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T12:43:06.129 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T12:43:06.129 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:06.129 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.133+0000 7f8a19f28640 1 librados: starting msgr at 2024-05-15T12:43:06.129 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.133+0000 7f8a19f28640 1 librados: starting objecter 2024-05-15T12:43:06.130 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.134+0000 7f8a19f28640 1 Processor -- start 2024-05-15T12:43:06.130 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.134+0000 7f8a19f28640 1 -- start start 2024-05-15T12:43:06.130 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.134+0000 7f8a19f28640 1 librados: setting wanted keys 2024-05-15T12:43:06.130 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.134+0000 7f8a19f28640 1 librados: calling monclient init 2024-05-15T12:43:06.130 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.134+0000 7f8a19f28640 10 monclient: init 2024-05-15T12:43:06.130 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.134+0000 7f8a19f28640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:06.130 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.134+0000 7f8a19f28640 10 monclient: _add_conns ranks=[0,2,1] 2024-05-15T12:43:06.130 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.134+0000 7f8a19f28640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8a140b5180 0x7f8a14138500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:06.130 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.134+0000 7f8a19f28640 10 monclient(hunting): picked mon.noname-a con 0x7f8a140b5180 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T12:43:06.130 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.134+0000 7f8a19f28640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8a1413ff40 0x7f8a14137830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:06.130 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.134+0000 7f8a19f28640 10 monclient(hunting): picked mon.noname-c con 0x7f8a1413ff40 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:43:06.130 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.134+0000 7f8a19f28640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8a14142d90 0x7f8a140acbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:06.130 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.134+0000 7f8a19f28640 10 monclient(hunting): picked mon.noname-b con 0x7f8a14142d90 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:43:06.130 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.134+0000 7f8a19f28640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:06.130 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.134+0000 7f8a19f28640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a1414e100 con 0x7f8a140b5180 2024-05-15T12:43:06.131 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.134+0000 7f8a19f28640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:06.131 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.134+0000 7f8a19f28640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a1414df80 con 0x7f8a14142d90 2024-05-15T12:43:06.131 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.134+0000 7f8a19f28640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:06.131 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.134+0000 7f8a19f28640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a1414e280 con 0x7f8a1413ff40 2024-05-15T12:43:06.131 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.134+0000 7f8a19f28640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:06.131 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.134+0000 7f8a19f28640 10 monclient(hunting): authenticate will time out at 1349.376714s 2024-05-15T12:43:06.131 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.135+0000 7f8a12ffd640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8a1413ff40 0x7f8a14137830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:06.131 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.135+0000 7f8a13fff640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8a14142d90 0x7f8a140acbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:06.131 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.135+0000 7f8a12ffd640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8a1413ff40 0x7f8a14137830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.117:48034/0 (socket says 172.21.15.117:48034) 2024-05-15T12:43:06.131 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.135+0000 7f8a12ffd640 1 -- 172.21.15.117:0/1718984491 learned_addr learned my addr 172.21.15.117:0/1718984491 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:43:06.131 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.135+0000 7f8a12ffd640 10 monclient(hunting): get_auth_request con 0x7f8a1413ff40 auth_method 0 2024-05-15T12:43:06.131 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.135+0000 7f8a12ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:06.131 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.135+0000 7f8a12ffd640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:06.131 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.135+0000 7f8a12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:06.131 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.135+0000 7f8a13fff640 10 monclient(hunting): get_auth_request con 0x7f8a14142d90 auth_method 0 2024-05-15T12:43:06.132 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.135+0000 7f8a13fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:06.132 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.135+0000 7f8a13fff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:06.132 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.135+0000 7f8a13fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:06.132 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.136+0000 7f8a13fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:06.132 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.136+0000 7f8a13fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:06.132 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.136+0000 7f8a13fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:06.132 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.136+0000 7f8a12ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:06.132 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.136+0000 7f8a12ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:06.132 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.136+0000 7f8a12ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:06.133 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.138+0000 7f8a13fff640 10 monclient(hunting): handle_auth_done global_id 4288 payload 824 2024-05-15T12:43:06.133 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.138+0000 7f8a137fe640 1 --2- 172.21.15.117:0/1718984491 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8a140b5180 0x7f8a14138500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:06.134 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.138+0000 7f8a13fff640 1 -- 172.21.15.117:0/1718984491 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8a1413ff40 msgr2=0x7f8a14137830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:06.134 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.138+0000 7f8a13fff640 1 --2- 172.21.15.117:0/1718984491 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8a1413ff40 0x7f8a14137830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:06.134 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.139+0000 7f8a13fff640 1 -- 172.21.15.117:0/1718984491 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8a140b5180 msgr2=0x7f8a14138500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:06.134 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.139+0000 7f8a13fff640 1 --2- 172.21.15.117:0/1718984491 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8a140b5180 0x7f8a14138500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:06.134 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.139+0000 7f8a13fff640 10 monclient: _finish_hunting 0 2024-05-15T12:43:06.134 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.139+0000 7f8a13fff640 1 monclient: found mon.noname-b 2024-05-15T12:43:06.134 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.139+0000 7f8a13fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:06.134 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.139+0000 7f8a13fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2024-05-15T12:43:06.134 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.139+0000 7f8a13fff640 1 -- 172.21.15.117:0/1718984491 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a14137d70 con 0x7f8a14142d90 2024-05-15T12:43:06.134 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.141+0000 7f8a13fff640 1 --2- 172.21.15.117:0/1718984491 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8a14142d90 0x7f8a140acbf0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8a140b5a00 tx=0x7f8a0c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:06.134 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.141+0000 7f8a0bfff640 10 client.?.objecter ms_handle_connect 0x7f8a14142d90 2024-05-15T12:43:06.134 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.141+0000 7f8a0bfff640 10 client.?.objecter resend_mon_ops 2024-05-15T12:43:06.183 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.192+0000 7f8a0bfff640 1 -- 172.21.15.117:0/1718984491 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a0c040070 con 0x7f8a14142d90 2024-05-15T12:43:06.183 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.192+0000 7f8a0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:06.183 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.192+0000 7f8a0bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:43:06.183 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.192+0000 7f8a0bfff640 10 monclient: dump: 2024-05-15T12:43:06.183 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:06.184 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:06.184 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:06.184 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:06.184 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:06.184 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:06.184 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:06.184 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:06.184 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:06.184 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:06.184 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.192+0000 7f8a0bfff640 10 monclient: _finish_auth 0 2024-05-15T12:43:06.184 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.192+0000 7f8a0bfff640 10 monclient: _check_auth_tickets 2024-05-15T12:43:06.184 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.192+0000 7f8a0bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:06.184 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.192+0000 7f8a0bfff640 1 -- 172.21.15.117:0/1718984491 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a0c030ca0 con 0x7f8a14142d90 2024-05-15T12:43:06.184 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.192+0000 7f8a19f28640 5 monclient: authenticate success, global_id 4288 2024-05-15T12:43:06.184 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.192+0000 7f8a0bfff640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:06.184 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.192+0000 7f8a0bfff640 1 -- 172.21.15.117:0/1718984491 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a0c035040 con 0x7f8a14142d90 2024-05-15T12:43:06.185 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.192+0000 7f8a19f28640 10 monclient: _renew_subs 2024-05-15T12:43:06.185 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.192+0000 7f8a19f28640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:43:06.185 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.192+0000 7f8a19f28640 1 -- 172.21.15.117:0/1718984491 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a140ac7d0 con 0x7f8a14142d90 2024-05-15T12:43:06.185 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.192+0000 7f8a0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:06.185 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.192+0000 7f8a0bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:43:06.185 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.192+0000 7f8a0bfff640 10 monclient: dump: 2024-05-15T12:43:06.185 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:06.185 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:06.185 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:06.185 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:06.185 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:06.185 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:06.185 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:06.185 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:06.185 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:06.185 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:06.185 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.192+0000 7f8a11ffb640 4 set_mon_vals no callback set 2024-05-15T12:43:06.186 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.192+0000 7f8a19f28640 10 client.4288.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:43:06.186 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.192+0000 7f8a19f28640 10 monclient: _renew_subs 2024-05-15T12:43:06.186 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.192+0000 7f8a19f28640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:43:06.186 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.192+0000 7f8a19f28640 1 -- 172.21.15.117:0/1718984491 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a14136490 con 0x7f8a14142d90 2024-05-15T12:43:06.186 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.192+0000 7f8a19f28640 1 librados: init done 2024-05-15T12:43:06.186 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.193+0000 7f8a19f28640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:06.186 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.193+0000 7f8a19f28640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:06.186 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.193+0000 7f8a19f28640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:43:06.186 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.193+0000 7f8a19f28640 1 -- 172.21.15.117:0/1718984491 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a1404f0e0 con 0x7f8a14142d90 2024-05-15T12:43:06.425 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.435+0000 7f8a0bfff640 1 -- 172.21.15.117:0/1718984491 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f8a0c04b060 con 0x7f8a14142d90 2024-05-15T12:43:06.425 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.435+0000 7f8a0bfff640 1 --2- 172.21.15.117:0/1718984491 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f89f406ceb0 0x7f89f406f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:06.426 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.435+0000 7f8a137fe640 1 --2- 172.21.15.117:0/1718984491 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f89f406ceb0 0x7f89f406f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:06.426 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.435+0000 7f8a137fe640 10 monclient: get_auth_request con 0x7f89f406ceb0 auth_method 0 2024-05-15T12:43:06.426 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.436+0000 7f8a137fe640 1 --2- 172.21.15.117:0/1718984491 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f89f406ceb0 0x7f89f406f370 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8a04001460 tx=0x7f8a04009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:06.426 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.436+0000 7f8a0bfff640 10 client.4288.objecter ms_handle_connect 0x7f89f406ceb0 2024-05-15T12:43:06.427 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.437+0000 7f8a0bfff640 1 -- 172.21.15.117:0/1718984491 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8a0c080f10 con 0x7f8a14142d90 2024-05-15T12:43:06.427 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.437+0000 7f8a0bfff640 10 client.4288.objecter ms_dispatch 0x7f8a14000b90 osd_map(22..22 src has 1..22) v4 2024-05-15T12:43:06.427 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.437+0000 7f8a0bfff640 3 client.4288.objecter handle_osd_map got epochs [22,22] > 0 2024-05-15T12:43:06.427 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.437+0000 7f8a0bfff640 3 client.4288.objecter handle_osd_map decoding full epoch 22 2024-05-15T12:43:06.427 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.437+0000 7f8a0bfff640 20 client.4288.objecter dump_active .. 0 homeless 2024-05-15T12:43:06.437 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.447+0000 7f8a0bfff640 1 -- 172.21.15.117:0/1718984491 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f8a0c02d040 con 0x7f8a14142d90 2024-05-15T12:43:06.437 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.447+0000 7f8a0bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:06.437 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.447+0000 7f8a0bfff640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:43:06.617 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.626+0000 7f8a19f28640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-05-15T12:43:06.617 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.626+0000 7f8a19f28640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-05-15T12:43:06.617 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.626+0000 7f8a19f28640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:43:06.617 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:06.626+0000 7f8a19f28640 1 -- 172.21.15.117:0/1718984491 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f8a14141cb0 con 0x7f8a14142d90 2024-05-15T12:43:07.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.221+0000 7f8a0bfff640 1 -- 172.21.15.117:0/1718984491 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v23) v1 ==== 109+0+0 (secure 0 0 0) 0x7f8a0c07b240 con 0x7f8a14142d90 2024-05-15T12:43:07.212 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.221+0000 7f8a0bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-05-15T12:43:07.212 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.221+0000 7f8a0bfff640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-05-15T12:43:07.212 INFO:teuthology.orchestra.run.smithi117.stderr:pool 'rbd' created 2024-05-15T12:43:07.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.224+0000 7f8a19f28640 10 librados: watch_flush enter 2024-05-15T12:43:07.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.224+0000 7f8a19f28640 10 librados: watch_flush exit 2024-05-15T12:43:07.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.224+0000 7f8a19f28640 20 client.4288.objecter shutdown clearing up homeless session... 2024-05-15T12:43:07.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.224+0000 7f8a19f28640 10 client.4288.objecter successfully canceled tick 2024-05-15T12:43:07.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.224+0000 7f8a19f28640 1 -- 172.21.15.117:0/1718984491 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f89f406ceb0 msgr2=0x7f89f406f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:07.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.224+0000 7f8a19f28640 1 --2- 172.21.15.117:0/1718984491 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f89f406ceb0 0x7f89f406f370 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8a04001460 tx=0x7f8a04009040 comp rx=0 tx=0).stop 2024-05-15T12:43:07.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.224+0000 7f8a19f28640 10 monclient: shutdown 2024-05-15T12:43:07.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.224+0000 7f8a19f28640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:07.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.224+0000 7f8a19f28640 1 -- 172.21.15.117:0/1718984491 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8a14142d90 msgr2=0x7f8a140acbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:07.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.224+0000 7f8a19f28640 1 --2- 172.21.15.117:0/1718984491 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8a14142d90 0x7f8a140acbf0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8a140b5a00 tx=0x7f8a0c0046c0 comp rx=0 tx=0).stop 2024-05-15T12:43:07.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.224+0000 7f8a19f28640 1 -- 172.21.15.117:0/1718984491 shutdown_connections 2024-05-15T12:43:07.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.224+0000 7f8a19f28640 1 --2- 172.21.15.117:0/1718984491 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f89f406ceb0 0x7f89f406f370 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:07.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.224+0000 7f8a19f28640 1 --2- 172.21.15.117:0/1718984491 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8a14142d90 0x7f8a140acbf0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:07.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.224+0000 7f8a19f28640 1 --2- 172.21.15.117:0/1718984491 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8a1413ff40 0x7f8a14137830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:07.216 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.224+0000 7f8a19f28640 1 --2- 172.21.15.117:0/1718984491 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8a140b5180 0x7f8a14138500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:07.216 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.224+0000 7f8a19f28640 1 -- 172.21.15.117:0/1718984491 >> 172.21.15.117:0/1718984491 conn(0x7f8a14066ed0 msgr2=0x7f8a14139610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:07.216 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.224+0000 7f8a19f28640 1 -- 172.21.15.117:0/1718984491 shutdown_connections 2024-05-15T12:43:07.216 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.224+0000 7f8a19f28640 1 -- 172.21.15.117:0/1718984491 wait complete. 2024-05-15T12:43:07.216 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.224+0000 7f8a19f28640 1 librados: shutdown 2024-05-15T12:43:07.228 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-05-15T12:43:07.346 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.355+0000 7f54ddcbe640 10 monclient: get_monmap_and_config 2024-05-15T12:43:07.346 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.355+0000 7f54ddcbe640 10 monclient: build_initial_monmap 2024-05-15T12:43:07.346 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.355+0000 7f54ddcbe640 10 monclient: monmap: 2024-05-15T12:43:07.346 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:07.346 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:07.346 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:43:07.357014+0000 2024-05-15T12:43:07.346 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:43:07.357014+0000 2024-05-15T12:43:07.346 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:07.346 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:07.346 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T12:43:07.346 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T12:43:07.346 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T12:43:07.346 INFO:teuthology.orchestra.run.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T12:43:07.347 INFO:teuthology.orchestra.run.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T12:43:07.347 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:07.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.356+0000 7f54ddcbe640 1 Processor -- start 2024-05-15T12:43:07.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.356+0000 7f54ddcbe640 1 -- start start 2024-05-15T12:43:07.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.356+0000 7f54ddcbe640 10 monclient: init 2024-05-15T12:43:07.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.357+0000 7f54ddcbe640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:07.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.357+0000 7f54ddcbe640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-05-15T12:43:07.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.357+0000 7f54ddcbe640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f54d80b86e0 0x7f54d80aa0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:07.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.357+0000 7f54ddcbe640 10 monclient(hunting): picked mon.noname-b con 0x7f54d80b86e0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:43:07.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.357+0000 7f54ddcbe640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f54d80b1dd0 0x7f54d8145bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:07.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.357+0000 7f54ddcbe640 10 monclient(hunting): picked mon.noname-a con 0x7f54d80b1dd0 addr v2:172.21.15.117:3300/0 2024-05-15T12:43:07.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.357+0000 7f54ddcbe640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54d813f870 0x7f54d813fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:07.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.357+0000 7f54ddcbe640 10 monclient(hunting): picked mon.noname-c con 0x7f54d813f870 addr v2:172.21.15.178:3300/0 2024-05-15T12:43:07.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.357+0000 7f54ddcbe640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:07.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.357+0000 7f54ddcbe640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54d80b8ac0 con 0x7f54d80b1dd0 2024-05-15T12:43:07.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.357+0000 7f54ddcbe640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:07.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.357+0000 7f54ddcbe640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54d80aa620 con 0x7f54d813f870 2024-05-15T12:43:07.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.357+0000 7f54ddcbe640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:07.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.357+0000 7f54ddcbe640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54d80a9bc0 con 0x7f54d80b86e0 2024-05-15T12:43:07.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.357+0000 7f54ddcbe640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:07.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.357+0000 7f54ddcbe640 10 monclient(hunting): authenticate will time out at 1350.599365s 2024-05-15T12:43:07.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.358+0000 7f54cffff640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f54d80b1dd0 0x7f54d8145bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:07.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.358+0000 7f54d77fe640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f54d80b86e0 0x7f54d80aa0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:07.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.358+0000 7f54d7fff640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54d813f870 0x7f54d813fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:07.351 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.358+0000 7f54d7fff640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54d813f870 0x7f54d813fcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:42762/0 (socket says 172.21.15.117:42762) 2024-05-15T12:43:07.351 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.358+0000 7f54d77fe640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f54d80b86e0 0x7f54d80aa0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.117:48044/0 (socket says 172.21.15.117:48044) 2024-05-15T12:43:07.351 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.358+0000 7f54cffff640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f54d80b1dd0 0x7f54d8145bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:47744/0 (socket says 172.21.15.117:47744) 2024-05-15T12:43:07.351 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.358+0000 7f54d77fe640 1 -- 172.21.15.117:0/2100500087 learned_addr learned my addr 172.21.15.117:0/2100500087 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:43:07.351 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.358+0000 7f54d77fe640 10 monclient(hunting): get_auth_request con 0x7f54d80b86e0 auth_method 0 2024-05-15T12:43:07.351 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.358+0000 7f54d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:07.351 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.358+0000 7f54d77fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:07.351 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.358+0000 7f54d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:07.351 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.358+0000 7f54d7fff640 10 monclient(hunting): get_auth_request con 0x7f54d813f870 auth_method 0 2024-05-15T12:43:07.351 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.358+0000 7f54d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:07.351 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.358+0000 7f54d7fff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:07.351 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.358+0000 7f54d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:07.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.358+0000 7f54cffff640 10 monclient(hunting): get_auth_request con 0x7f54d80b1dd0 auth_method 0 2024-05-15T12:43:07.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.358+0000 7f54cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:07.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.358+0000 7f54cffff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:07.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.358+0000 7f54cffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:07.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.359+0000 7f54d7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:07.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.359+0000 7f54d7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:07.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.359+0000 7f54d7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:07.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.359+0000 7f54cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:07.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.359+0000 7f54cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:07.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.359+0000 7f54cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:07.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.359+0000 7f54d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:07.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.359+0000 7f54d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:07.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.359+0000 7f54d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:07.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.360+0000 7f54d7fff640 10 monclient(hunting): handle_auth_done global_id 4291 payload 274 2024-05-15T12:43:07.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.360+0000 7f54d7fff640 1 -- 172.21.15.117:0/2100500087 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f54d80b86e0 msgr2=0x7f54d80aa0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:43:07.353 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.360+0000 7f54d7fff640 1 --2- 172.21.15.117:0/2100500087 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f54d80b86e0 0x7f54d80aa0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:07.353 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.360+0000 7f54d7fff640 1 -- 172.21.15.117:0/2100500087 >> v2:172.21.15.117:3300/0 conn(0x7f54d80b1dd0 msgr2=0x7f54d8145bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:43:07.353 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.360+0000 7f54d7fff640 1 --2- 172.21.15.117:0/2100500087 >> v2:172.21.15.117:3300/0 conn(0x7f54d80b1dd0 0x7f54d8145bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:07.353 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.360+0000 7f54d7fff640 10 monclient: _finish_hunting 0 2024-05-15T12:43:07.353 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.360+0000 7f54d7fff640 1 monclient: found mon.noname-c 2024-05-15T12:43:07.353 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.360+0000 7f54d7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:07.353 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.360+0000 7f54d7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.178:3300/0 2024-05-15T12:43:07.353 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.360+0000 7f54d7fff640 1 -- 172.21.15.117:0/2100500087 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54d8146100 con 0x7f54d813f870 2024-05-15T12:43:07.353 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.362+0000 7f54d7fff640 1 --2- 172.21.15.117:0/2100500087 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54d813f870 0x7f54d813fcc0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f54d80aaff0 tx=0x7f54d002bd90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c3d15462b8d0190 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:07.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.366+0000 7f54d6ffd640 1 -- 172.21.15.117:0/2100500087 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d0039070 con 0x7f54d813f870 2024-05-15T12:43:07.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.366+0000 7f54d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:07.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.366+0000 7f54d6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:43:07.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.366+0000 7f54d6ffd640 10 monclient: dump: 2024-05-15T12:43:07.358 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:07.358 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:07.358 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:07.358 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:07.358 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:07.358 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:07.358 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:07.358 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:07.359 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:07.359 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:07.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.366+0000 7f54d6ffd640 10 monclient: _finish_auth 0 2024-05-15T12:43:07.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.366+0000 7f54d6ffd640 10 monclient: _check_auth_tickets 2024-05-15T12:43:07.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.366+0000 7f54d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:07.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.366+0000 7f54d6ffd640 1 -- 172.21.15.117:0/2100500087 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54d002cb40 con 0x7f54d813f870 2024-05-15T12:43:07.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.366+0000 7f54d6ffd640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:07.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.366+0000 7f54d6ffd640 1 -- 172.21.15.117:0/2100500087 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d0034440 con 0x7f54d813f870 2024-05-15T12:43:07.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.366+0000 7f54d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:07.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.366+0000 7f54d6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:43:07.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.366+0000 7f54d6ffd640 10 monclient: dump: 2024-05-15T12:43:07.359 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:07.359 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:07.359 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:07.359 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:07.360 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:07.360 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:07.360 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:07.360 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:07.360 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:07.360 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:07.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.366+0000 7f54ddcbe640 5 monclient: authenticate success, global_id 4291 2024-05-15T12:43:07.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.366+0000 7f54ddcbe640 10 monclient: get_monmap_and_config success 2024-05-15T12:43:07.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.366+0000 7f54ddcbe640 4 set_mon_vals no callback set 2024-05-15T12:43:07.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.367+0000 7f54ddcbe640 10 monclient: shutdown 2024-05-15T12:43:07.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.367+0000 7f54ddcbe640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:07.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.367+0000 7f54ddcbe640 1 -- 172.21.15.117:0/2100500087 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54d813f870 msgr2=0x7f54d813fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:07.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.367+0000 7f54ddcbe640 1 --2- 172.21.15.117:0/2100500087 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54d813f870 0x7f54d813fcc0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f54d80aaff0 tx=0x7f54d002bd90 comp rx=0 tx=0).stop 2024-05-15T12:43:07.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.367+0000 7f54ddcbe640 1 -- 172.21.15.117:0/2100500087 shutdown_connections 2024-05-15T12:43:07.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.367+0000 7f54ddcbe640 1 --2- 172.21.15.117:0/2100500087 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54d813f870 0x7f54d813fcc0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:07.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.367+0000 7f54ddcbe640 1 --2- 172.21.15.117:0/2100500087 >> v2:172.21.15.117:3300/0 conn(0x7f54d80b1dd0 0x7f54d8145bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:07.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.367+0000 7f54ddcbe640 1 --2- 172.21.15.117:0/2100500087 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f54d80b86e0 0x7f54d80aa0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:07.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.367+0000 7f54ddcbe640 1 -- 172.21.15.117:0/2100500087 >> 172.21.15.117:0/2100500087 conn(0x7f54d8066f00 msgr2=0x7f54d8067310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:07.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.367+0000 7f54ddcbe640 1 -- 172.21.15.117:0/2100500087 shutdown_connections 2024-05-15T12:43:07.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.367+0000 7f54ddcbe640 1 -- 172.21.15.117:0/2100500087 wait complete. 2024-05-15T12:43:07.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.367+0000 7f54ddcbe640 10 monclient: build_initial_monmap 2024-05-15T12:43:07.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.367+0000 7f54ddcbe640 10 monclient: monmap: 2024-05-15T12:43:07.361 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:07.361 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:43:07.361 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-05-15T12:43:07.361 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-05-15T12:43:07.361 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:07.361 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:07.361 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T12:43:07.361 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T12:43:07.362 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T12:43:07.362 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:07.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.368+0000 7f54ddcbe640 1 librados: starting msgr at 2024-05-15T12:43:07.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.368+0000 7f54ddcbe640 1 librados: starting objecter 2024-05-15T12:43:07.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.368+0000 7f54ddcbe640 1 Processor -- start 2024-05-15T12:43:07.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.368+0000 7f54ddcbe640 1 -- start start 2024-05-15T12:43:07.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.368+0000 7f54ddcbe640 1 librados: setting wanted keys 2024-05-15T12:43:07.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.368+0000 7f54ddcbe640 1 librados: calling monclient init 2024-05-15T12:43:07.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.368+0000 7f54ddcbe640 10 monclient: init 2024-05-15T12:43:07.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.368+0000 7f54ddcbe640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:07.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.368+0000 7f54ddcbe640 10 monclient: _add_conns ranks=[1,0,2] 2024-05-15T12:43:07.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.368+0000 7f54ddcbe640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54d80b1dd0 0x7f54d8142f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:07.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.368+0000 7f54ddcbe640 10 monclient(hunting): picked mon.noname-b con 0x7f54d80b1dd0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:43:07.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.368+0000 7f54ddcbe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54d80b86e0 0x7f54d80b0e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:07.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.368+0000 7f54ddcbe640 10 monclient(hunting): picked mon.noname-a con 0x7f54d80b86e0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T12:43:07.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.368+0000 7f54ddcbe640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f54d813f870 0x7f54d80b1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:07.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.368+0000 7f54ddcbe640 10 monclient(hunting): picked mon.noname-c con 0x7f54d813f870 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:43:07.366 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.368+0000 7f54ddcbe640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:07.366 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.368+0000 7f54ddcbe640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54d80b3a60 con 0x7f54d80b86e0 2024-05-15T12:43:07.366 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.368+0000 7f54ddcbe640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:07.366 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.368+0000 7f54ddcbe640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54d80a9850 con 0x7f54d80b1dd0 2024-05-15T12:43:07.366 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.368+0000 7f54ddcbe640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:07.366 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.368+0000 7f54ddcbe640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54d80b3be0 con 0x7f54d813f870 2024-05-15T12:43:07.366 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.368+0000 7f54ddcbe640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:07.366 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.368+0000 7f54ddcbe640 10 monclient(hunting): authenticate will time out at 1350.610525s 2024-05-15T12:43:07.366 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.369+0000 7f54d7fff640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f54d813f870 0x7f54d80b1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:07.366 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.369+0000 7f54cffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54d80b86e0 0x7f54d80b0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:07.366 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.369+0000 7f54d77fe640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54d80b1dd0 0x7f54d8142f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:07.366 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.369+0000 7f54cffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54d80b86e0 0x7f54d80b0e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:47760/0 (socket says 172.21.15.117:47760) 2024-05-15T12:43:07.367 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.369+0000 7f54d7fff640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f54d813f870 0x7f54d80b1350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.117:48058/0 (socket says 172.21.15.117:48058) 2024-05-15T12:43:07.367 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.369+0000 7f54cffff640 1 -- 172.21.15.117:0/923108333 learned_addr learned my addr 172.21.15.117:0/923108333 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:43:07.367 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.369+0000 7f54cffff640 10 monclient(hunting): get_auth_request con 0x7f54d80b86e0 auth_method 0 2024-05-15T12:43:07.367 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.369+0000 7f54cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:07.367 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.369+0000 7f54cffff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:07.367 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.369+0000 7f54cffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:07.367 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.369+0000 7f54d7fff640 10 monclient(hunting): get_auth_request con 0x7f54d813f870 auth_method 0 2024-05-15T12:43:07.367 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.369+0000 7f54d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:07.367 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.369+0000 7f54d7fff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:07.367 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.369+0000 7f54d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:07.367 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.369+0000 7f54d77fe640 10 monclient(hunting): get_auth_request con 0x7f54d80b1dd0 auth_method 0 2024-05-15T12:43:07.367 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.369+0000 7f54d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:07.367 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.369+0000 7f54d77fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:07.367 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.369+0000 7f54d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:07.367 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.370+0000 7f54d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:07.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.370+0000 7f54d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:07.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.370+0000 7f54d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:07.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.370+0000 7f54d7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:07.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.370+0000 7f54d7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:07.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.370+0000 7f54d7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:07.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.370+0000 7f54cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:07.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.370+0000 7f54cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:07.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.370+0000 7f54cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:07.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.372+0000 7f54d77fe640 10 monclient(hunting): handle_auth_done global_id 4294 payload 824 2024-05-15T12:43:07.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.372+0000 7f54d77fe640 1 -- 172.21.15.117:0/923108333 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f54d813f870 msgr2=0x7f54d80b1350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:07.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.372+0000 7f54d77fe640 1 --2- 172.21.15.117:0/923108333 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f54d813f870 0x7f54d80b1350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:07.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.372+0000 7f54d77fe640 1 -- 172.21.15.117:0/923108333 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54d80b86e0 msgr2=0x7f54d80b0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:07.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.372+0000 7f54d77fe640 1 --2- 172.21.15.117:0/923108333 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54d80b86e0 0x7f54d80b0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:07.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.372+0000 7f54d77fe640 10 monclient: _finish_hunting 0 2024-05-15T12:43:07.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.372+0000 7f54d77fe640 1 monclient: found mon.noname-b 2024-05-15T12:43:07.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.372+0000 7f54d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:07.369 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.372+0000 7f54d77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2024-05-15T12:43:07.369 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.372+0000 7f54d77fe640 1 -- 172.21.15.117:0/923108333 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54d80ad140 con 0x7f54d80b1dd0 2024-05-15T12:43:07.369 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.374+0000 7f54d77fe640 1 --2- 172.21.15.117:0/923108333 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54d80b1dd0 0x7f54d8142f70 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f54c800e850 tx=0x7f54c800ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:07.369 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.374+0000 7f54d4ff9640 10 client.?.objecter ms_handle_connect 0x7f54d80b1dd0 2024-05-15T12:43:07.369 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.374+0000 7f54d4ff9640 10 client.?.objecter resend_mon_ops 2024-05-15T12:43:07.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.378+0000 7f54d4ff9640 1 -- 172.21.15.117:0/923108333 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c8017070 con 0x7f54d80b1dd0 2024-05-15T12:43:07.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.378+0000 7f54d4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:07.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.378+0000 7f54d4ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:43:07.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.378+0000 7f54d4ff9640 10 monclient: dump: 2024-05-15T12:43:07.371 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:07.371 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:07.371 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:07.371 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:07.371 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:07.371 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:07.371 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:07.371 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:07.371 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:07.371 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:07.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.378+0000 7f54d4ff9640 10 monclient: _finish_auth 0 2024-05-15T12:43:07.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.378+0000 7f54d4ff9640 10 monclient: _check_auth_tickets 2024-05-15T12:43:07.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.378+0000 7f54d4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:07.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.378+0000 7f54d4ff9640 1 -- 172.21.15.117:0/923108333 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54c800c940 con 0x7f54d80b1dd0 2024-05-15T12:43:07.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.378+0000 7f54ddcbe640 5 monclient: authenticate success, global_id 4294 2024-05-15T12:43:07.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.378+0000 7f54ddcbe640 10 monclient: _renew_subs 2024-05-15T12:43:07.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.378+0000 7f54ddcbe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:43:07.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.378+0000 7f54ddcbe640 1 -- 172.21.15.117:0/923108333 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54d80ad3b0 con 0x7f54d80b1dd0 2024-05-15T12:43:07.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.378+0000 7f54d4ff9640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:07.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.378+0000 7f54d4ff9640 1 -- 172.21.15.117:0/923108333 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c800ccf0 con 0x7f54d80b1dd0 2024-05-15T12:43:07.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.378+0000 7f54d4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:07.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.378+0000 7f54d4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:43:07.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.378+0000 7f54d4ff9640 10 monclient: dump: 2024-05-15T12:43:07.372 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:07.372 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:07.372 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:07.372 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:07.372 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:07.372 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:07.372 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:07.373 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:07.373 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:07.373 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:07.373 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.378+0000 7f54ddcbe640 10 client.4294.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:43:07.373 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.378+0000 7f54d67fc640 4 set_mon_vals no callback set 2024-05-15T12:43:07.373 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.378+0000 7f54ddcbe640 10 monclient: _renew_subs 2024-05-15T12:43:07.373 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.378+0000 7f54ddcbe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:43:07.373 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.378+0000 7f54ddcbe640 1 -- 172.21.15.117:0/923108333 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54d80b1890 con 0x7f54d80b1dd0 2024-05-15T12:43:07.373 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.378+0000 7f54ddcbe640 1 librados: init done 2024-05-15T12:43:07.373 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.379+0000 7f54ddcbe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:07.373 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.379+0000 7f54ddcbe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:07.373 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.379+0000 7f54ddcbe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:43:07.373 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.379+0000 7f54ddcbe640 1 -- 172.21.15.117:0/923108333 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54d804f0e0 con 0x7f54d80b1dd0 2024-05-15T12:43:07.386 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.397+0000 7f54d4ff9640 1 -- 172.21.15.117:0/923108333 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f54c801a070 con 0x7f54d80b1dd0 2024-05-15T12:43:07.387 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.397+0000 7f54d4ff9640 1 --2- 172.21.15.117:0/923108333 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f54b806ceb0 0x7f54b806f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:07.387 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.397+0000 7f54cffff640 1 --2- 172.21.15.117:0/923108333 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f54b806ceb0 0x7f54b806f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:07.387 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.397+0000 7f54cffff640 10 monclient: get_auth_request con 0x7f54b806ceb0 auth_method 0 2024-05-15T12:43:07.387 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.397+0000 7f54d4ff9640 10 client.4294.objecter ms_handle_connect 0x7f54b806ceb0 2024-05-15T12:43:07.387 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.397+0000 7f54cffff640 1 --2- 172.21.15.117:0/923108333 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f54b806ceb0 0x7f54b806f370 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f54d813d3d0 tx=0x7f54c0009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:07.390 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.400+0000 7f54d4ff9640 1 -- 172.21.15.117:0/923108333 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5199+0+0 (secure 0 0 0) 0x7f54c8057100 con 0x7f54d80b1dd0 2024-05-15T12:43:07.390 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.400+0000 7f54d4ff9640 10 client.4294.objecter ms_dispatch 0x7f54d8000b90 osd_map(23..23 src has 1..23) v4 2024-05-15T12:43:07.390 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.400+0000 7f54d4ff9640 3 client.4294.objecter handle_osd_map got epochs [23,23] > 0 2024-05-15T12:43:07.390 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.400+0000 7f54d4ff9640 3 client.4294.objecter handle_osd_map decoding full epoch 23 2024-05-15T12:43:07.390 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.400+0000 7f54d4ff9640 20 client.4294.objecter dump_active .. 0 homeless 2024-05-15T12:43:07.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.665+0000 7f54d4ff9640 1 -- 172.21.15.117:0/923108333 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f54d80ad3b0 con 0x7f54d80b1dd0 2024-05-15T12:43:07.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.665+0000 7f54d4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:07.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.665+0000 7f54d4ff9640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:43:07.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.859+0000 7f54ddcbe640 10 start_mon_command cmd=[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-05-15T12:43:07.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.859+0000 7f54ddcbe640 10 monclient: _send_command 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-05-15T12:43:07.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.859+0000 7f54ddcbe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:43:07.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:07.859+0000 7f54ddcbe640 1 -- 172.21.15.117:0/923108333 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f54d813be90 con 0x7f54d80b1dd0 2024-05-15T12:43:08.432 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.441+0000 7f54d4ff9640 1 -- 172.21.15.117:0/923108333 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v24) v1 ==== 173+0+0 (secure 0 0 0) 0x7f54c8059060 con 0x7f54d80b1dd0 2024-05-15T12:43:08.432 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.441+0000 7f54d4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-05-15T12:43:08.432 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.441+0000 7f54d4ff9640 10 monclient: _finish_command 2 = system:0 enabled application 'rbd' on pool 'rbd' 2024-05-15T12:43:08.432 INFO:teuthology.orchestra.run.smithi117.stderr:enabled application 'rbd' on pool 'rbd' 2024-05-15T12:43:08.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.444+0000 7f54ddcbe640 10 librados: watch_flush enter 2024-05-15T12:43:08.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.444+0000 7f54ddcbe640 10 librados: watch_flush exit 2024-05-15T12:43:08.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.444+0000 7f54ddcbe640 20 client.4294.objecter shutdown clearing up homeless session... 2024-05-15T12:43:08.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.444+0000 7f54ddcbe640 10 client.4294.objecter successfully canceled tick 2024-05-15T12:43:08.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.444+0000 7f54ddcbe640 1 -- 172.21.15.117:0/923108333 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f54b806ceb0 msgr2=0x7f54b806f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:08.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.444+0000 7f54ddcbe640 1 --2- 172.21.15.117:0/923108333 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f54b806ceb0 0x7f54b806f370 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f54d813d3d0 tx=0x7f54c0009190 comp rx=0 tx=0).stop 2024-05-15T12:43:08.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.444+0000 7f54ddcbe640 10 monclient: shutdown 2024-05-15T12:43:08.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.444+0000 7f54ddcbe640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:08.436 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.444+0000 7f54ddcbe640 1 -- 172.21.15.117:0/923108333 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54d80b1dd0 msgr2=0x7f54d8142f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:08.436 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.444+0000 7f54ddcbe640 1 --2- 172.21.15.117:0/923108333 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54d80b1dd0 0x7f54d8142f70 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f54c800e850 tx=0x7f54c800ed10 comp rx=0 tx=0).stop 2024-05-15T12:43:08.436 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.444+0000 7f54ddcbe640 1 -- 172.21.15.117:0/923108333 shutdown_connections 2024-05-15T12:43:08.436 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.444+0000 7f54ddcbe640 1 --2- 172.21.15.117:0/923108333 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f54d813f870 0x7f54d80b1350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:08.436 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.444+0000 7f54ddcbe640 1 --2- 172.21.15.117:0/923108333 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54d80b86e0 0x7f54d80b0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:08.436 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.444+0000 7f54ddcbe640 1 --2- 172.21.15.117:0/923108333 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f54b806ceb0 0x7f54b806f370 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:08.436 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.444+0000 7f54ddcbe640 1 --2- 172.21.15.117:0/923108333 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54d80b1dd0 0x7f54d8142f70 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:08.436 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.444+0000 7f54ddcbe640 1 -- 172.21.15.117:0/923108333 >> 172.21.15.117:0/923108333 conn(0x7f54d8066f00 msgr2=0x7f54d80b4fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:08.436 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.444+0000 7f54ddcbe640 1 -- 172.21.15.117:0/923108333 shutdown_connections 2024-05-15T12:43:08.436 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.444+0000 7f54ddcbe640 1 -- 172.21.15.117:0/923108333 wait complete. 2024-05-15T12:43:08.436 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.444+0000 7f54ddcbe640 1 librados: shutdown 2024-05-15T12:43:08.446 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-05-15T12:43:08.447 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-05-15T12:43:08.447 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-05-15T12:43:08.447 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-05-15T12:43:08.447 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-05-15T12:43:08.546 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.556+0000 7f2295afa640 10 monclient: get_monmap_and_config 2024-05-15T12:43:08.546 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.556+0000 7f2295afa640 10 monclient: build_initial_monmap 2024-05-15T12:43:08.546 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.556+0000 7f2295afa640 10 monclient: monmap: 2024-05-15T12:43:08.546 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:08.546 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:08.546 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:43:08.557159+0000 2024-05-15T12:43:08.547 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:43:08.557159+0000 2024-05-15T12:43:08.547 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:08.547 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:08.547 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T12:43:08.547 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T12:43:08.547 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T12:43:08.547 INFO:teuthology.orchestra.run.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T12:43:08.547 INFO:teuthology.orchestra.run.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T12:43:08.547 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:08.550 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.556+0000 7f2295afa640 1 Processor -- start 2024-05-15T12:43:08.550 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.556+0000 7f2295afa640 1 -- start start 2024-05-15T12:43:08.550 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.556+0000 7f2295afa640 10 monclient: init 2024-05-15T12:43:08.550 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.557+0000 7f2295afa640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:08.551 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.557+0000 7f2295afa640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-05-15T12:43:08.551 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.557+0000 7f2295afa640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f22900b4d00 0x7f22900b1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:08.551 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.557+0000 7f2295afa640 10 monclient(hunting): picked mon.noname-a con 0x7f22900b4d00 addr v2:172.21.15.117:3300/0 2024-05-15T12:43:08.551 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.557+0000 7f2295afa640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f22900b50e0 0x7f2290145c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:08.551 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.557+0000 7f2295afa640 10 monclient(hunting): picked mon.noname-b con 0x7f22900b50e0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:43:08.551 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.557+0000 7f2295afa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f22900a94b0 addr v1:172.21.15.117:6789/0 2024-05-15T12:43:08.551 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.557+0000 7f2295afa640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22900ab1b0 con 0x7f22900a94b0 2024-05-15T12:43:08.552 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.557+0000 7f2295afa640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:08.552 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.557+0000 7f2295afa640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22900b23c0 con 0x7f22900b4d00 2024-05-15T12:43:08.552 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.557+0000 7f2295afa640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:08.552 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.557+0000 7f2295afa640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22900b16f0 con 0x7f22900b50e0 2024-05-15T12:43:08.552 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.557+0000 7f2295afa640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:08.552 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.557+0000 7f2295afa640 10 monclient(hunting): authenticate will time out at 1351.799357s 2024-05-15T12:43:08.552 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.558+0000 7f228f7fe640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f22900b4d00 0x7f22900b1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:08.552 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.558+0000 7f228effd640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f22900b50e0 0x7f2290145c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:08.552 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.558+0000 7f228ffff640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f22900a94b0 0x7f22900b0fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:50554/0 (socket says 172.21.15.117:50554) 2024-05-15T12:43:08.552 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.558+0000 7f228ffff640 1 -- 172.21.15.117:0/3537956727 learned_addr learned my addr 172.21.15.117:0/3537956727 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-05-15T12:43:08.552 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.558+0000 7f228effd640 10 monclient(hunting): get_auth_request con 0x7f22900b50e0 auth_method 0 2024-05-15T12:43:08.552 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.558+0000 7f228effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:08.552 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.558+0000 7f228effd640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:08.552 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.558+0000 7f228effd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:08.553 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.558+0000 7f228f7fe640 10 monclient(hunting): get_auth_request con 0x7f22900b4d00 auth_method 0 2024-05-15T12:43:08.553 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.558+0000 7f228f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:08.553 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.558+0000 7f228f7fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:08.553 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.558+0000 7f228f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:08.553 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.559+0000 7f228effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:08.553 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.559+0000 7f228effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:08.553 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.559+0000 7f228effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:08.553 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.559+0000 7f228f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:08.553 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.559+0000 7f228f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:08.553 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.559+0000 7f228f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:08.554 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.561+0000 7f228effd640 10 monclient(hunting): handle_auth_done global_id 4250 payload 274 2024-05-15T12:43:08.554 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.561+0000 7f228effd640 1 -- 172.21.15.117:0/3537956727 >> v2:172.21.15.117:3300/0 conn(0x7f22900b4d00 msgr2=0x7f22900b1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:43:08.554 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.561+0000 7f228effd640 1 --2- 172.21.15.117:0/3537956727 >> v2:172.21.15.117:3300/0 conn(0x7f22900b4d00 0x7f22900b1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:08.554 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.561+0000 7f228effd640 1 -- 172.21.15.117:0/3537956727 >> v1:172.21.15.117:6789/0 conn(0x7f22900a94b0 legacy=0x7f22900b0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:08.554 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.561+0000 7f228effd640 10 monclient: _finish_hunting 0 2024-05-15T12:43:08.554 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.561+0000 7f228effd640 1 monclient: found mon.noname-b 2024-05-15T12:43:08.555 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.561+0000 7f228effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:08.555 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.561+0000 7f228effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.117:3301/0 2024-05-15T12:43:08.555 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.561+0000 7f228effd640 1 -- 172.21.15.117:0/3537956727 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2290146160 con 0x7f22900b50e0 2024-05-15T12:43:08.555 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.563+0000 7f228effd640 1 --2- 172.21.15.117:0/3537956727 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f22900b50e0 0x7f2290145c20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f22840097a0 tx=0x7f228402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c2746511c5821b8 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:08.556 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.565+0000 7f228e7fc640 1 -- 172.21.15.117:0/3537956727 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22840315a0 con 0x7f22900b50e0 2024-05-15T12:43:08.556 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.565+0000 7f228e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:08.556 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.565+0000 7f228e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:43:08.556 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.565+0000 7f228e7fc640 10 monclient: dump: 2024-05-15T12:43:08.556 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:08.556 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:08.556 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:08.556 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:08.556 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:08.556 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:08.556 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:08.556 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:08.556 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:08.556 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:08.557 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.565+0000 7f228e7fc640 10 monclient: _finish_auth 0 2024-05-15T12:43:08.557 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.565+0000 7f228e7fc640 10 monclient: _check_auth_tickets 2024-05-15T12:43:08.557 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.565+0000 7f228e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:08.557 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.565+0000 7f2295afa640 5 monclient: authenticate success, global_id 4250 2024-05-15T12:43:08.557 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.565+0000 7f2295afa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:43:08.557 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.567+0000 7f228e7fc640 1 -- 172.21.15.117:0/3537956727 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2284037070 con 0x7f22900b50e0 2024-05-15T12:43:08.558 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.567+0000 7f228e7fc640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:08.558 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.567+0000 7f228e7fc640 1 -- 172.21.15.117:0/3537956727 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2284031960 con 0x7f22900b50e0 2024-05-15T12:43:08.558 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.567+0000 7f2295afa640 10 monclient: get_monmap_and_config success 2024-05-15T12:43:08.558 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.567+0000 7f2295afa640 4 set_mon_vals no callback set 2024-05-15T12:43:08.558 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.567+0000 7f228e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:08.558 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.567+0000 7f228e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T12:43:08.558 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.567+0000 7f228e7fc640 10 monclient: dump: 2024-05-15T12:43:08.558 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:08.558 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:08.558 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:08.558 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:08.558 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:08.558 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:08.558 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:08.559 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:08.559 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:08.559 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:08.562 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.568+0000 7f2295afa640 10 monclient: shutdown 2024-05-15T12:43:08.562 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.568+0000 7f2295afa640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:08.563 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.568+0000 7f2295afa640 1 -- 172.21.15.117:0/3537956727 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f22900b50e0 msgr2=0x7f2290145c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:08.563 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.568+0000 7f2295afa640 1 --2- 172.21.15.117:0/3537956727 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f22900b50e0 0x7f2290145c20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f22840097a0 tx=0x7f228402b840 comp rx=0 tx=0).stop 2024-05-15T12:43:08.563 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.568+0000 7f2295afa640 1 -- 172.21.15.117:0/3537956727 shutdown_connections 2024-05-15T12:43:08.563 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.568+0000 7f2295afa640 1 --2- 172.21.15.117:0/3537956727 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f22900b50e0 0x7f2290145c20 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:08.563 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.568+0000 7f2295afa640 1 --2- 172.21.15.117:0/3537956727 >> v2:172.21.15.117:3300/0 conn(0x7f22900b4d00 0x7f22900b1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:08.563 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.568+0000 7f2295afa640 1 -- 172.21.15.117:0/3537956727 >> 172.21.15.117:0/3537956727 conn(0x7f2290066cc0 msgr2=0x7f22900670d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:08.563 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.568+0000 7f2295afa640 1 -- 172.21.15.117:0/3537956727 shutdown_connections 2024-05-15T12:43:08.563 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.568+0000 7f2295afa640 1 -- 172.21.15.117:0/3537956727 wait complete. 2024-05-15T12:43:08.563 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.569+0000 7f2295afa640 10 monclient: build_initial_monmap 2024-05-15T12:43:08.563 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.569+0000 7f2295afa640 10 monclient: monmap: 2024-05-15T12:43:08.563 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:08.563 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:43:08.563 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-05-15T12:43:08.563 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-05-15T12:43:08.564 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:08.564 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:08.564 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T12:43:08.564 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T12:43:08.564 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T12:43:08.564 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:08.564 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.569+0000 7f2295afa640 1 librados: starting msgr at 2024-05-15T12:43:08.564 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.569+0000 7f2295afa640 1 librados: starting objecter 2024-05-15T12:43:08.564 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.569+0000 7f2295afa640 1 Processor -- start 2024-05-15T12:43:08.564 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.569+0000 7f2295afa640 1 -- start start 2024-05-15T12:43:08.564 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.569+0000 7f2295afa640 1 librados: setting wanted keys 2024-05-15T12:43:08.564 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.569+0000 7f2295afa640 1 librados: calling monclient init 2024-05-15T12:43:08.564 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.569+0000 7f2295afa640 10 monclient: init 2024-05-15T12:43:08.564 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.570+0000 7f2295afa640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:08.564 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.570+0000 7f2295afa640 10 monclient: _add_conns ranks=[2,0,1] 2024-05-15T12:43:08.565 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.570+0000 7f2295afa640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f22900a94b0 0x7f22900abbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:08.565 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.570+0000 7f2295afa640 10 monclient(hunting): picked mon.noname-c con 0x7f22900a94b0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:43:08.565 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.570+0000 7f2295afa640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f22900b4d00 0x7f22900ac740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:08.565 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.570+0000 7f2295afa640 10 monclient(hunting): picked mon.noname-a con 0x7f22900b4d00 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T12:43:08.565 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.570+0000 7f2295afa640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f22901d7e60 0x7f22901d82f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:08.565 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.570+0000 7f2295afa640 10 monclient(hunting): picked mon.noname-b con 0x7f22901d7e60 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:43:08.565 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.570+0000 7f2295afa640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:08.565 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.570+0000 7f2295afa640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f229014cfb0 con 0x7f22900b4d00 2024-05-15T12:43:08.565 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.570+0000 7f2295afa640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:08.565 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.570+0000 7f2295afa640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f229014d130 con 0x7f22901d7e60 2024-05-15T12:43:08.565 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.570+0000 7f2295afa640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:08.565 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.570+0000 7f2295afa640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f229014d2b0 con 0x7f22900a94b0 2024-05-15T12:43:08.565 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.570+0000 7f2295afa640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:08.565 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.570+0000 7f2295afa640 10 monclient(hunting): authenticate will time out at 1351.812348s 2024-05-15T12:43:08.566 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.571+0000 7f228f7fe640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f22900a94b0 0x7f22900abbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:08.566 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.571+0000 7f228effd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f22900b4d00 0x7f22900ac740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:08.566 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.571+0000 7f228ffff640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f22901d7e60 0x7f22901d82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:08.566 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.571+0000 7f228f7fe640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f22900a94b0 0x7f22900abbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.117:48076/0 (socket says 172.21.15.117:48076) 2024-05-15T12:43:08.566 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.571+0000 7f228f7fe640 1 -- 172.21.15.117:0/3376011023 learned_addr learned my addr 172.21.15.117:0/3376011023 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:43:08.566 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.571+0000 7f228f7fe640 10 monclient(hunting): get_auth_request con 0x7f22900a94b0 auth_method 0 2024-05-15T12:43:08.566 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.571+0000 7f228f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:08.566 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.571+0000 7f228f7fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:08.566 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.571+0000 7f228f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:08.566 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.571+0000 7f228effd640 10 monclient(hunting): get_auth_request con 0x7f22900b4d00 auth_method 0 2024-05-15T12:43:08.566 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.571+0000 7f228effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:08.566 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.571+0000 7f228effd640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:08.566 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.571+0000 7f228effd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:08.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.571+0000 7f228ffff640 10 monclient(hunting): get_auth_request con 0x7f22901d7e60 auth_method 0 2024-05-15T12:43:08.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.571+0000 7f228ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:08.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.571+0000 7f228ffff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:08.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.571+0000 7f228ffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:08.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.572+0000 7f228effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:08.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.572+0000 7f228effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:08.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.572+0000 7f228effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:08.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.572+0000 7f228ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:08.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.572+0000 7f228ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:08.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.572+0000 7f228ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:08.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.572+0000 7f228f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:08.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.572+0000 7f228f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:08.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.572+0000 7f228f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:08.569 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.574+0000 7f228effd640 10 monclient(hunting): handle_auth_done global_id 4272 payload 824 2024-05-15T12:43:08.569 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.574+0000 7f228effd640 1 -- 172.21.15.117:0/3376011023 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f22900a94b0 msgr2=0x7f22900abbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:08.569 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.574+0000 7f228effd640 1 --2- 172.21.15.117:0/3376011023 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f22900a94b0 0x7f22900abbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:08.569 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.575+0000 7f228effd640 1 -- 172.21.15.117:0/3376011023 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f22901d7e60 msgr2=0x7f22901d82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:08.569 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.575+0000 7f228effd640 1 --2- 172.21.15.117:0/3376011023 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f22901d7e60 0x7f22901d82f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:08.569 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.575+0000 7f228effd640 10 monclient: _finish_hunting 0 2024-05-15T12:43:08.569 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.575+0000 7f228effd640 1 monclient: found mon.noname-a 2024-05-15T12:43:08.570 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.575+0000 7f228effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:08.570 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.575+0000 7f228effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.117:3300/0 2024-05-15T12:43:08.570 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.575+0000 7f228effd640 1 -- 172.21.15.117:0/3376011023 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22901d7980 con 0x7f22900b4d00 2024-05-15T12:43:08.570 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.575+0000 7f228f7fe640 1 --2- 172.21.15.117:0/3376011023 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f22900a94b0 0x7f22900abbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-15T12:43:08.570 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.577+0000 7f228effd640 1 --2- 172.21.15.117:0/3376011023 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f22900b4d00 0x7f22900ac740 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2284033000 tx=0x7f22840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:08.570 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.577+0000 7f2277fff640 10 client.?.objecter ms_handle_connect 0x7f22900b4d00 2024-05-15T12:43:08.570 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.577+0000 7f2277fff640 10 client.?.objecter resend_mon_ops 2024-05-15T12:43:08.573 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.582+0000 7f2277fff640 1 -- 172.21.15.117:0/3376011023 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2284032630 con 0x7f22900b4d00 2024-05-15T12:43:08.573 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.582+0000 7f2277fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:08.573 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.582+0000 7f2277fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T12:43:08.573 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.582+0000 7f2277fff640 10 monclient: dump: 2024-05-15T12:43:08.573 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:08.573 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:08.573 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:08.573 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:08.573 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:08.573 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:08.574 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:08.574 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:08.574 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:08.574 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:08.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.582+0000 7f2277fff640 10 monclient: _finish_auth 0 2024-05-15T12:43:08.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.582+0000 7f2277fff640 10 monclient: _check_auth_tickets 2024-05-15T12:43:08.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.582+0000 7f2277fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:08.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.582+0000 7f2277fff640 1 -- 172.21.15.117:0/3376011023 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22840327d0 con 0x7f22900b4d00 2024-05-15T12:43:08.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.582+0000 7f2295afa640 5 monclient: authenticate success, global_id 4272 2024-05-15T12:43:08.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.582+0000 7f2295afa640 10 monclient: _renew_subs 2024-05-15T12:43:08.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.582+0000 7f2295afa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:43:08.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.582+0000 7f2295afa640 1 -- 172.21.15.117:0/3376011023 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22900adaa0 con 0x7f22900b4d00 2024-05-15T12:43:08.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.582+0000 7f2277fff640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:08.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.582+0000 7f2277fff640 1 -- 172.21.15.117:0/3376011023 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2284032ba0 con 0x7f22900b4d00 2024-05-15T12:43:08.575 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.582+0000 7f2277fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:08.575 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.582+0000 7f2277fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T12:43:08.575 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.582+0000 7f2277fff640 10 monclient: dump: 2024-05-15T12:43:08.575 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:08.575 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:08.575 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:08.575 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:08.575 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:08.575 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:08.575 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:08.575 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:08.575 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:08.575 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:08.575 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.582+0000 7f228dffb640 4 set_mon_vals no callback set 2024-05-15T12:43:08.575 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.582+0000 7f2295afa640 10 client.4272.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:43:08.576 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.582+0000 7f2295afa640 10 monclient: _renew_subs 2024-05-15T12:43:08.576 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.582+0000 7f2295afa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:43:08.576 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.582+0000 7f2295afa640 1 -- 172.21.15.117:0/3376011023 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22901420a0 con 0x7f22900b4d00 2024-05-15T12:43:08.576 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.582+0000 7f2295afa640 1 librados: init done 2024-05-15T12:43:08.576 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.583+0000 7f2295afa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:08.576 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.583+0000 7f2295afa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:08.576 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.583+0000 7f2295afa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:43:08.576 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.583+0000 7f2295afa640 1 -- 172.21.15.117:0/3376011023 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f229004f0e0 con 0x7f22900b4d00 2024-05-15T12:43:08.840 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.849+0000 7f2277fff640 1 -- 172.21.15.117:0/3376011023 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f228404e430 con 0x7f22900b4d00 2024-05-15T12:43:08.840 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.850+0000 7f2277fff640 1 --2- 172.21.15.117:0/3376011023 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f227c06ce60 0x7f227c06f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:08.840 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.850+0000 7f228f7fe640 1 --2- 172.21.15.117:0/3376011023 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f227c06ce60 0x7f227c06f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:08.840 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.850+0000 7f228f7fe640 10 monclient: get_auth_request con 0x7f227c06ce60 auth_method 0 2024-05-15T12:43:08.840 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.850+0000 7f2277fff640 10 client.4272.objecter ms_handle_connect 0x7f227c06ce60 2024-05-15T12:43:08.840 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.850+0000 7f228f7fe640 1 --2- 172.21.15.117:0/3376011023 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f227c06ce60 0x7f227c06f320 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f22800112c0 tx=0x7f2280009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:08.854 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.864+0000 7f2277fff640 1 -- 172.21.15.117:0/3376011023 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f228402f030 con 0x7f22900b4d00 2024-05-15T12:43:08.854 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.864+0000 7f2277fff640 10 client.4272.objecter ms_dispatch 0x7f2290000b90 osd_map(24..24 src has 1..24) v4 2024-05-15T12:43:08.855 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.864+0000 7f2277fff640 3 client.4272.objecter handle_osd_map got epochs [24,24] > 0 2024-05-15T12:43:08.855 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.864+0000 7f2277fff640 3 client.4272.objecter handle_osd_map decoding full epoch 24 2024-05-15T12:43:08.855 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.864+0000 7f2277fff640 20 client.4272.objecter dump_active .. 0 homeless 2024-05-15T12:43:08.855 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.864+0000 7f2277fff640 1 -- 172.21.15.117:0/3376011023 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f2284040070 con 0x7f22900b4d00 2024-05-15T12:43:08.855 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.864+0000 7f2277fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:08.855 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:08.864+0000 7f2277fff640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:43:09.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.098+0000 7f2295afa640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-05-15T12:43:09.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.098+0000 7f2295afa640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-05-15T12:43:09.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.098+0000 7f2295afa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:43:09.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.098+0000 7f2295afa640 1 -- 172.21.15.117:0/3376011023 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f22901d7b50 con 0x7f22900b4d00 2024-05-15T12:43:09.233 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.242+0000 7f2277fff640 1 -- 172.21.15.117:0/3376011023 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+171788 (secure 0 0 0) 0x7f228407aa40 con 0x7f22900b4d00 2024-05-15T12:43:09.233 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.242+0000 7f2277fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-05-15T12:43:09.233 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.242+0000 7f2277fff640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:43:09.235 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:43:09.237 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.245+0000 7f2295afa640 10 librados: watch_flush enter 2024-05-15T12:43:09.237 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.245+0000 7f2295afa640 10 librados: watch_flush exit 2024-05-15T12:43:09.237 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.245+0000 7f2295afa640 20 client.4272.objecter shutdown clearing up homeless session... 2024-05-15T12:43:09.237 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.245+0000 7f2295afa640 10 client.4272.objecter successfully canceled tick 2024-05-15T12:43:09.238 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.245+0000 7f2295afa640 1 -- 172.21.15.117:0/3376011023 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f227c06ce60 msgr2=0x7f227c06f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:09.238 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.245+0000 7f2295afa640 1 --2- 172.21.15.117:0/3376011023 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f227c06ce60 0x7f227c06f320 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f22800112c0 tx=0x7f2280009190 comp rx=0 tx=0).stop 2024-05-15T12:43:09.238 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.245+0000 7f2295afa640 10 monclient: shutdown 2024-05-15T12:43:09.238 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.245+0000 7f2295afa640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:09.238 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.245+0000 7f2295afa640 1 -- 172.21.15.117:0/3376011023 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f22900b4d00 msgr2=0x7f22900ac740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:09.238 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.245+0000 7f2295afa640 1 --2- 172.21.15.117:0/3376011023 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f22900b4d00 0x7f22900ac740 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2284033000 tx=0x7f22840046c0 comp rx=0 tx=0).stop 2024-05-15T12:43:09.238 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.246+0000 7f2295afa640 1 -- 172.21.15.117:0/3376011023 shutdown_connections 2024-05-15T12:43:09.238 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.246+0000 7f2295afa640 1 --2- 172.21.15.117:0/3376011023 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f227c06ce60 0x7f227c06f320 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:09.238 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.246+0000 7f2295afa640 1 --2- 172.21.15.117:0/3376011023 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f22901d7e60 0x7f22901d82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:09.238 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.246+0000 7f2295afa640 1 --2- 172.21.15.117:0/3376011023 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f22900b4d00 0x7f22900ac740 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:09.238 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.246+0000 7f2295afa640 1 --2- 172.21.15.117:0/3376011023 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f22900a94b0 0x7f22900abbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:09.238 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.246+0000 7f2295afa640 1 -- 172.21.15.117:0/3376011023 >> 172.21.15.117:0/3376011023 conn(0x7f2290066cc0 msgr2=0x7f229013b660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:09.238 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.246+0000 7f2295afa640 1 -- 172.21.15.117:0/3376011023 shutdown_connections 2024-05-15T12:43:09.238 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.246+0000 7f2295afa640 1 -- 172.21.15.117:0/3376011023 wait complete. 2024-05-15T12:43:09.239 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.246+0000 7f2295afa640 1 librados: shutdown 2024-05-15T12:43:09.247 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":5,"flags":0,"active_gid":4107,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1785766523},{"type":"v1","addr":"172.21.15.117:6801","nonce":1785766523}]},"active_addr":"172.21.15.117:6801/1785766523","active_change":"2024-05-15T12:42:09.633151+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4114,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.117:0","nonce":223813219}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.117:0","nonce":3646927167}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.117:0","nonce":447301358}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.117:0","nonce":4219456208}]}]} 2024-05-15T12:43:09.253 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-05-15T12:43:09.253 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-05-15T12:43:09.253 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-15T12:43:09.355 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.365+0000 7fe121da1640 10 monclient: get_monmap_and_config 2024-05-15T12:43:09.355 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.365+0000 7fe121da1640 10 monclient: build_initial_monmap 2024-05-15T12:43:09.355 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.365+0000 7fe121da1640 10 monclient: monmap: 2024-05-15T12:43:09.355 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:09.355 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:09.355 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:43:09.366236+0000 2024-05-15T12:43:09.355 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:43:09.366236+0000 2024-05-15T12:43:09.356 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:09.356 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:09.356 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T12:43:09.356 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T12:43:09.356 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T12:43:09.356 INFO:teuthology.orchestra.run.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T12:43:09.356 INFO:teuthology.orchestra.run.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T12:43:09.356 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:09.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.366+0000 7fe121da1640 1 Processor -- start 2024-05-15T12:43:09.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.366+0000 7fe121da1640 1 -- start start 2024-05-15T12:43:09.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.366+0000 7fe121da1640 10 monclient: init 2024-05-15T12:43:09.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.367+0000 7fe121da1640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:09.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.367+0000 7fe121da1640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-05-15T12:43:09.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.367+0000 7fe121da1640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fe11c0b4d00 0x7fe11c0b1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:09.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.367+0000 7fe121da1640 10 monclient(hunting): picked mon.noname-c con 0x7fe11c0b4d00 addr v2:172.21.15.178:3300/0 2024-05-15T12:43:09.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.367+0000 7fe121da1640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe11c0b50e0 0x7fe11c145c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:09.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.367+0000 7fe121da1640 10 monclient(hunting): picked mon.noname-b con 0x7fe11c0b50e0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:43:09.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.367+0000 7fe121da1640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fe11c0a94b0 0x7fe11c0b0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:09.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.367+0000 7fe121da1640 10 monclient(hunting): picked mon.noname-a con 0x7fe11c0a94b0 addr v2:172.21.15.117:3300/0 2024-05-15T12:43:09.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.367+0000 7fe121da1640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:09.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.367+0000 7fe121da1640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe11c0b23c0 con 0x7fe11c0a94b0 2024-05-15T12:43:09.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.367+0000 7fe121da1640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:09.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.367+0000 7fe121da1640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe11c0ab1b0 con 0x7fe11c0b4d00 2024-05-15T12:43:09.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.367+0000 7fe121da1640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:09.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.367+0000 7fe121da1640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe11c146160 con 0x7fe11c0b50e0 2024-05-15T12:43:09.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.367+0000 7fe121da1640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:09.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.367+0000 7fe121da1640 10 monclient(hunting): authenticate will time out at 1352.609183s 2024-05-15T12:43:09.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.367+0000 7fe11b7fe640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fe11c0b4d00 0x7fe11c0b1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:09.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.368+0000 7fe11b7fe640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fe11c0b4d00 0x7fe11c0b1e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:42804/0 (socket says 172.21.15.117:42804) 2024-05-15T12:43:09.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.368+0000 7fe11b7fe640 1 -- 172.21.15.117:0/1985040362 learned_addr learned my addr 172.21.15.117:0/1985040362 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:43:09.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.368+0000 7fe11b7fe640 10 monclient(hunting): get_auth_request con 0x7fe11c0b4d00 auth_method 0 2024-05-15T12:43:09.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.368+0000 7fe11b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:09.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.368+0000 7fe11b7fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:09.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.368+0000 7fe11b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:09.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.368+0000 7fe11affd640 1 --2- 172.21.15.117:0/1985040362 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe11c0b50e0 0x7fe11c145c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:09.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.368+0000 7fe11b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:09.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.368+0000 7fe11b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:09.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.368+0000 7fe11b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:09.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.368+0000 7fe11affd640 10 monclient(hunting): get_auth_request con 0x7fe11c0b50e0 auth_method 0 2024-05-15T12:43:09.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.368+0000 7fe11affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:09.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.368+0000 7fe11affd640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:09.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.368+0000 7fe11affd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:09.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.369+0000 7fe11affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:09.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.369+0000 7fe11affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:09.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.369+0000 7fe11affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:09.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.370+0000 7fe11b7fe640 10 monclient(hunting): handle_auth_done global_id 4300 payload 274 2024-05-15T12:43:09.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.370+0000 7fe11b7fe640 1 -- 172.21.15.117:0/1985040362 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe11c0b50e0 msgr2=0x7fe11c145c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:43:09.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.370+0000 7fe11b7fe640 1 --2- 172.21.15.117:0/1985040362 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe11c0b50e0 0x7fe11c145c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:09.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.370+0000 7fe11b7fe640 1 -- 172.21.15.117:0/1985040362 >> v2:172.21.15.117:3300/0 conn(0x7fe11c0a94b0 msgr2=0x7fe11c0b0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:43:09.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.370+0000 7fe11b7fe640 1 --2- 172.21.15.117:0/1985040362 >> v2:172.21.15.117:3300/0 conn(0x7fe11c0a94b0 0x7fe11c0b0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:09.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.370+0000 7fe11b7fe640 10 monclient: _finish_hunting 0 2024-05-15T12:43:09.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.370+0000 7fe11b7fe640 1 monclient: found mon.noname-c 2024-05-15T12:43:09.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.370+0000 7fe11b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:09.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.370+0000 7fe11b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.178:3300/0 2024-05-15T12:43:09.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.370+0000 7fe11b7fe640 1 -- 172.21.15.117:0/1985040362 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe11c0b1520 con 0x7fe11c0b4d00 2024-05-15T12:43:09.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.372+0000 7fe11b7fe640 1 --2- 172.21.15.117:0/1985040362 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe11c0b4d00 0x7fe11c0b1e80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe11c0b3f10 tx=0x7fe10c02b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f6b3b829473af29 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:09.366 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.376+0000 7fe11a7fc640 1 -- 172.21.15.117:0/1985040362 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe10c02e070 con 0x7fe11c0b4d00 2024-05-15T12:43:09.366 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.376+0000 7fe11a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:09.366 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.376+0000 7fe11a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:43:09.366 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.376+0000 7fe11a7fc640 10 monclient: dump: 2024-05-15T12:43:09.366 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:09.366 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:09.366 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:09.366 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:09.366 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:09.367 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:09.367 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:09.367 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:09.367 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:09.367 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:09.367 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.376+0000 7fe11a7fc640 10 monclient: _finish_auth 0 2024-05-15T12:43:09.367 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.376+0000 7fe11a7fc640 10 monclient: _check_auth_tickets 2024-05-15T12:43:09.367 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.376+0000 7fe11a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:09.367 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.376+0000 7fe11a7fc640 1 -- 172.21.15.117:0/1985040362 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe10c037070 con 0x7fe11c0b4d00 2024-05-15T12:43:09.367 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.376+0000 7fe121da1640 5 monclient: authenticate success, global_id 4300 2024-05-15T12:43:09.367 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.376+0000 7fe121da1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:43:09.367 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.376+0000 7fe11a7fc640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:09.367 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.376+0000 7fe11a7fc640 1 -- 172.21.15.117:0/1985040362 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe10c031a00 con 0x7fe11c0b4d00 2024-05-15T12:43:09.367 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.376+0000 7fe11a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:09.367 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.376+0000 7fe11a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:43:09.367 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.376+0000 7fe11a7fc640 10 monclient: dump: 2024-05-15T12:43:09.368 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:09.368 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:09.368 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:09.368 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:09.368 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:09.368 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:09.368 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:09.368 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:09.368 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:09.368 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:09.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.376+0000 7fe121da1640 10 monclient: get_monmap_and_config success 2024-05-15T12:43:09.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.376+0000 7fe121da1640 4 set_mon_vals no callback set 2024-05-15T12:43:09.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.377+0000 7fe121da1640 10 monclient: shutdown 2024-05-15T12:43:09.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.377+0000 7fe121da1640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:09.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.377+0000 7fe121da1640 1 -- 172.21.15.117:0/1985040362 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe11c0b4d00 msgr2=0x7fe11c0b1e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:09.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.377+0000 7fe121da1640 1 --2- 172.21.15.117:0/1985040362 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe11c0b4d00 0x7fe11c0b1e80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe11c0b3f10 tx=0x7fe10c02b880 comp rx=0 tx=0).stop 2024-05-15T12:43:09.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.377+0000 7fe121da1640 1 -- 172.21.15.117:0/1985040362 shutdown_connections 2024-05-15T12:43:09.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.377+0000 7fe121da1640 1 --2- 172.21.15.117:0/1985040362 >> v2:172.21.15.117:3300/0 conn(0x7fe11c0a94b0 0x7fe11c0b0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:09.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.377+0000 7fe121da1640 1 --2- 172.21.15.117:0/1985040362 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe11c0b50e0 0x7fe11c145c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:09.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.377+0000 7fe121da1640 1 --2- 172.21.15.117:0/1985040362 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe11c0b4d00 0x7fe11c0b1e80 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:09.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.377+0000 7fe121da1640 1 -- 172.21.15.117:0/1985040362 >> 172.21.15.117:0/1985040362 conn(0x7fe11c066cc0 msgr2=0x7fe11c0670d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:09.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.377+0000 7fe121da1640 1 -- 172.21.15.117:0/1985040362 shutdown_connections 2024-05-15T12:43:09.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.377+0000 7fe121da1640 1 -- 172.21.15.117:0/1985040362 wait complete. 2024-05-15T12:43:09.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.378+0000 7fe121da1640 10 monclient: build_initial_monmap 2024-05-15T12:43:09.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.378+0000 7fe121da1640 10 monclient: monmap: 2024-05-15T12:43:09.372 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:09.372 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:43:09.372 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-05-15T12:43:09.372 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-05-15T12:43:09.372 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:09.372 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:09.372 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T12:43:09.373 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T12:43:09.373 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T12:43:09.373 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:09.373 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.378+0000 7fe121da1640 1 librados: starting msgr at 2024-05-15T12:43:09.373 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.378+0000 7fe121da1640 1 librados: starting objecter 2024-05-15T12:43:09.373 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.378+0000 7fe121da1640 1 Processor -- start 2024-05-15T12:43:09.373 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.379+0000 7fe121da1640 1 -- start start 2024-05-15T12:43:09.373 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.379+0000 7fe121da1640 1 librados: setting wanted keys 2024-05-15T12:43:09.373 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.379+0000 7fe121da1640 1 librados: calling monclient init 2024-05-15T12:43:09.373 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.379+0000 7fe121da1640 10 monclient: init 2024-05-15T12:43:09.373 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.379+0000 7fe121da1640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:09.373 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.379+0000 7fe121da1640 10 monclient: _add_conns ranks=[1,0,2] 2024-05-15T12:43:09.373 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.379+0000 7fe121da1640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe11c0a94b0 0x7fe11c1df730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:09.373 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.379+0000 7fe121da1640 10 monclient(hunting): picked mon.noname-b con 0x7fe11c0a94b0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:43:09.374 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.380+0000 7fe121da1640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe11c0b4d00 0x7fe11c13c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:09.374 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.380+0000 7fe121da1640 10 monclient(hunting): picked mon.noname-a con 0x7fe11c0b4d00 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T12:43:09.374 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.380+0000 7fe121da1640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe11c0b08d0 0x7fe11c13cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:09.374 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.380+0000 7fe121da1640 10 monclient(hunting): picked mon.noname-c con 0x7fe11c0b08d0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:43:09.374 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.380+0000 7fe121da1640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:09.374 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.380+0000 7fe121da1640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe11c142700 con 0x7fe11c0b4d00 2024-05-15T12:43:09.374 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.380+0000 7fe121da1640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:09.374 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.380+0000 7fe121da1640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe11c142880 con 0x7fe11c0a94b0 2024-05-15T12:43:09.374 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.380+0000 7fe121da1640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:09.374 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.380+0000 7fe121da1640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe11c142a00 con 0x7fe11c0b08d0 2024-05-15T12:43:09.374 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.380+0000 7fe121da1640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:09.374 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.380+0000 7fe121da1640 10 monclient(hunting): authenticate will time out at 1352.621850s 2024-05-15T12:43:09.376 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.382+0000 7fe11b7fe640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe11c0a94b0 0x7fe11c1df730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:09.376 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.383+0000 7fe11affd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe11c0b4d00 0x7fe11c13c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:09.376 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.384+0000 7fe11affd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe11c0b4d00 0x7fe11c13c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:47808/0 (socket says 172.21.15.117:47808) 2024-05-15T12:43:09.376 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.385+0000 7fe11affd640 1 -- 172.21.15.117:0/2611534735 learned_addr learned my addr 172.21.15.117:0/2611534735 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:43:09.376 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.385+0000 7fe11affd640 10 monclient(hunting): get_auth_request con 0x7fe11c0b4d00 auth_method 0 2024-05-15T12:43:09.376 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.385+0000 7fe11affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:09.376 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.385+0000 7fe11affd640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:09.377 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.385+0000 7fe11affd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:09.377 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.385+0000 7fe11affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:09.377 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.385+0000 7fe11affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:09.377 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.385+0000 7fe11affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:09.377 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.386+0000 7fe11bfff640 1 --2- 172.21.15.117:0/2611534735 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe11c0b08d0 0x7fe11c13cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:09.377 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.387+0000 7fe11bfff640 10 monclient(hunting): get_auth_request con 0x7fe11c0b08d0 auth_method 0 2024-05-15T12:43:09.377 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.387+0000 7fe11bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:09.377 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.387+0000 7fe11bfff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:09.378 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.387+0000 7fe11bfff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:09.378 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.387+0000 7fe11bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:09.378 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.387+0000 7fe11bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:09.378 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.387+0000 7fe11bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:09.378 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.388+0000 7fe11b7fe640 10 monclient(hunting): get_auth_request con 0x7fe11c0a94b0 auth_method 0 2024-05-15T12:43:09.378 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.388+0000 7fe11b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:09.378 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.388+0000 7fe11b7fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:09.378 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.388+0000 7fe11b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:09.379 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.389+0000 7fe11b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:09.379 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.389+0000 7fe11b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:09.379 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.389+0000 7fe11b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:09.380 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.390+0000 7fe11bfff640 10 monclient(hunting): handle_auth_done global_id 4259 payload 824 2024-05-15T12:43:09.380 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.390+0000 7fe11bfff640 1 -- 172.21.15.117:0/2611534735 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe11c0a94b0 msgr2=0x7fe11c1df730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:09.381 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.390+0000 7fe11bfff640 1 --2- 172.21.15.117:0/2611534735 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe11c0a94b0 0x7fe11c1df730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:09.381 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.390+0000 7fe11bfff640 1 -- 172.21.15.117:0/2611534735 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe11c0b4d00 msgr2=0x7fe11c13c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:09.381 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.390+0000 7fe11bfff640 1 --2- 172.21.15.117:0/2611534735 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe11c0b4d00 0x7fe11c13c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:09.381 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.390+0000 7fe11bfff640 10 monclient: _finish_hunting 0 2024-05-15T12:43:09.381 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.390+0000 7fe11bfff640 1 monclient: found mon.noname-c 2024-05-15T12:43:09.381 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.390+0000 7fe11bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:09.381 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.390+0000 7fe11bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.117:3301/0 2024-05-15T12:43:09.381 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.390+0000 7fe11bfff640 1 -- 172.21.15.117:0/2611534735 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe11c1dff70 con 0x7fe11c0b08d0 2024-05-15T12:43:09.383 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.393+0000 7fe11bfff640 1 --2- 172.21.15.117:0/2611534735 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe11c0b08d0 0x7fe11c13cf00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe11400be20 tx=0x7fe11400bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:09.383 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.393+0000 7fe103fff640 10 client.?.objecter ms_handle_connect 0x7fe11c0b08d0 2024-05-15T12:43:09.383 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.393+0000 7fe103fff640 10 client.?.objecter resend_mon_ops 2024-05-15T12:43:09.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.445+0000 7fe103fff640 1 -- 172.21.15.117:0/2611534735 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe114019070 con 0x7fe11c0b08d0 2024-05-15T12:43:09.436 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.445+0000 7fe103fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:09.436 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.445+0000 7fe103fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:43:09.436 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.445+0000 7fe103fff640 10 monclient: dump: 2024-05-15T12:43:09.436 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:09.436 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:09.436 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:09.436 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:09.436 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:09.436 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:09.436 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:09.436 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:09.436 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:09.436 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:09.436 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.445+0000 7fe103fff640 10 monclient: _finish_auth 0 2024-05-15T12:43:09.437 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.445+0000 7fe103fff640 10 monclient: _check_auth_tickets 2024-05-15T12:43:09.437 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.445+0000 7fe103fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:09.437 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.445+0000 7fe103fff640 1 -- 172.21.15.117:0/2611534735 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe11400ce10 con 0x7fe11c0b08d0 2024-05-15T12:43:09.437 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.445+0000 7fe103fff640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:09.437 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.445+0000 7fe103fff640 1 -- 172.21.15.117:0/2611534735 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe114014680 con 0x7fe11c0b08d0 2024-05-15T12:43:09.437 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.445+0000 7fe103fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:09.437 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.445+0000 7fe103fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T12:43:09.437 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.445+0000 7fe103fff640 10 monclient: dump: 2024-05-15T12:43:09.437 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:09.437 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:09.437 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:09.437 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:09.437 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:09.437 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:09.437 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:09.438 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:09.438 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:09.438 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:09.438 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.445+0000 7fe121da1640 5 monclient: authenticate success, global_id 4259 2024-05-15T12:43:09.438 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.445+0000 7fe121da1640 10 monclient: _renew_subs 2024-05-15T12:43:09.438 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.445+0000 7fe121da1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T12:43:09.438 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.445+0000 7fe121da1640 1 -- 172.21.15.117:0/2611534735 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe11c1dfc70 con 0x7fe11c0b08d0 2024-05-15T12:43:09.438 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.445+0000 7fe119ffb640 4 set_mon_vals no callback set 2024-05-15T12:43:09.438 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.445+0000 7fe121da1640 10 client.4259.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:43:09.438 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.445+0000 7fe121da1640 10 monclient: _renew_subs 2024-05-15T12:43:09.438 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.445+0000 7fe121da1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T12:43:09.438 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.445+0000 7fe121da1640 1 -- 172.21.15.117:0/2611534735 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe11c0ab9e0 con 0x7fe11c0b08d0 2024-05-15T12:43:09.438 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.445+0000 7fe121da1640 1 librados: init done 2024-05-15T12:43:09.439 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.446+0000 7fe121da1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:09.439 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.446+0000 7fe121da1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:09.439 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.446+0000 7fe121da1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T12:43:09.439 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.446+0000 7fe121da1640 1 -- 172.21.15.117:0/2611534735 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe11c04f0e0 con 0x7fe11c0b08d0 2024-05-15T12:43:09.457 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.468+0000 7fe103fff640 1 -- 172.21.15.117:0/2611534735 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fe11401c070 con 0x7fe11c0b08d0 2024-05-15T12:43:09.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.468+0000 7fe103fff640 1 --2- 172.21.15.117:0/2611534735 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fe10806ce90 0x7fe10806f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:09.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.468+0000 7fe11b7fe640 1 --2- 172.21.15.117:0/2611534735 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fe10806ce90 0x7fe10806f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:09.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.468+0000 7fe11b7fe640 10 monclient: get_auth_request con 0x7fe10806ce90 auth_method 0 2024-05-15T12:43:09.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.469+0000 7fe11b7fe640 1 --2- 172.21.15.117:0/2611534735 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fe10806ce90 0x7fe10806f350 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe10c036000 tx=0x7fe10c03d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:09.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.469+0000 7fe103fff640 10 client.4259.objecter ms_handle_connect 0x7fe10806ce90 2024-05-15T12:43:09.461 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.471+0000 7fe103fff640 1 -- 172.21.15.117:0/2611534735 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fe114056d70 con 0x7fe11c0b08d0 2024-05-15T12:43:09.461 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.471+0000 7fe103fff640 10 client.4259.objecter ms_dispatch 0x7fe11c000b90 osd_map(24..24 src has 1..24) v4 2024-05-15T12:43:09.461 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.471+0000 7fe103fff640 3 client.4259.objecter handle_osd_map got epochs [24,24] > 0 2024-05-15T12:43:09.461 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.471+0000 7fe103fff640 3 client.4259.objecter handle_osd_map decoding full epoch 24 2024-05-15T12:43:09.461 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.471+0000 7fe103fff640 20 client.4259.objecter dump_active .. 0 homeless 2024-05-15T12:43:09.717 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.727+0000 7fe103fff640 1 -- 172.21.15.117:0/2611534735 <== mon.2 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe114023020 con 0x7fe11c0b08d0 2024-05-15T12:43:09.717 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.727+0000 7fe103fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:09.717 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.727+0000 7fe103fff640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:43:09.928 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.938+0000 7fe121da1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:43:09.928 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.938+0000 7fe121da1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:43:09.928 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.938+0000 7fe121da1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T12:43:09.928 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.938+0000 7fe121da1640 1 -- 172.21.15.117:0/2611534735 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe11c1e0140 con 0x7fe11c0b08d0 2024-05-15T12:43:09.961 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.970+0000 7fe103fff640 1 -- 172.21.15.117:0/2611534735 <== mon.2 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+14790 (secure 0 0 0) 0x7fe114058cd0 con 0x7fe11c0b08d0 2024-05-15T12:43:09.961 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.970+0000 7fe103fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:43:09.961 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.970+0000 7fe103fff640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:43:09.961 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:43:09.962 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":25,"fsid":"8817f6cf-d933-4825-b972-575ef59f5ba9","created":"2024-05-15T12:42:04.264009+0000","modified":"2024-05-15T12:43:09.352925+0000","last_up_change":"2024-05-15T12:43:01.039161+0000","last_in_change":"2024-05-15T12:42:09.879357+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-15T12:42:52.250376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-05-15T12:43:06.641055+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aaffd6d9-1a00-4793-9d1f-65a30501bec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":1673738625},{"type":"v1","addr":"172.21.15.117:6827","nonce":1673738625}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":1673738625},{"type":"v1","addr":"172.21.15.117:6829","nonce":1673738625}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":1673738625},{"type":"v1","addr":"172.21.15.117:6833","nonce":1673738625}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":1673738625},{"type":"v1","addr":"172.21.15.117:6831","nonce":1673738625}]},"public_addr":"172.21.15.117:6827/1673738625","cluster_addr":"172.21.15.117:6829/1673738625","heartbeat_back_addr":"172.21.15.117:6833/1673738625","heartbeat_front_addr":"172.21.15.117:6831/1673738625","state":["exists","up"]},{"osd":1,"uuid":"efd4e8ed-699c-4ccb-b989-cfadc7aaed35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3864975175},{"type":"v1","addr":"172.21.15.117:6819","nonce":3864975175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3864975175},{"type":"v1","addr":"172.21.15.117:6821","nonce":3864975175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3864975175},{"type":"v1","addr":"172.21.15.117:6825","nonce":3864975175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3864975175},{"type":"v1","addr":"172.21.15.117:6823","nonce":3864975175}]},"public_addr":"172.21.15.117:6819/3864975175","cluster_addr":"172.21.15.117:6821/3864975175","heartbeat_back_addr":"172.21.15.117:6825/3864975175","heartbeat_front_addr":"172.21.15.117:6823/3864975175","state":["exists","up"]},{"osd":2,"uuid":"708e4e3e-7e79-480f-9dbf-7b972c2f2811","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1425835241},{"type":"v1","addr":"172.21.15.117:6805","nonce":1425835241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1425835241},{"type":"v1","addr":"172.21.15.117:6811","nonce":1425835241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":1425835241},{"type":"v1","addr":"172.21.15.117:6817","nonce":1425835241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6813","nonce":1425835241},{"type":"v1","addr":"172.21.15.117:6815","nonce":1425835241}]},"public_addr":"172.21.15.117:6805/1425835241","cluster_addr":"172.21.15.117:6811/1425835241","heartbeat_back_addr":"172.21.15.117:6817/1425835241","heartbeat_front_addr":"172.21.15.117:6815/1425835241","state":["exists","up"]},{"osd":3,"uuid":"e7b5d7d7-515c-42d9-9c41-b1ca3977865e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3842021926},{"type":"v1","addr":"172.21.15.117:6803","nonce":3842021926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3842021926},{"type":"v1","addr":"172.21.15.117:6807","nonce":3842021926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":3842021926},{"type":"v1","addr":"172.21.15.117:6814","nonce":3842021926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":3842021926},{"type":"v1","addr":"172.21.15.117:6809","nonce":3842021926}]},"public_addr":"172.21.15.117:6803/3842021926","cluster_addr":"172.21.15.117:6807/3842021926","heartbeat_back_addr":"172.21.15.117:6814/3842021926","heartbeat_front_addr":"172.21.15.117:6809/3842021926","state":["exists","up"]},{"osd":4,"uuid":"fa647c2c-942b-4be3-afb9-6dba6b0a8cf3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":248472851},{"type":"v1","addr":"172.21.15.178:6825","nonce":248472851}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":248472851},{"type":"v1","addr":"172.21.15.178:6827","nonce":248472851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":248472851},{"type":"v1","addr":"172.21.15.178:6831","nonce":248472851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":248472851},{"type":"v1","addr":"172.21.15.178:6829","nonce":248472851}]},"public_addr":"172.21.15.178:6825/248472851","cluster_addr":"172.21.15.178:6827/248472851","heartbeat_back_addr":"172.21.15.178:6831/248472851","heartbeat_front_addr":"172.21.15.178:6829/248472851","state":["exists","up"]},{"osd":5,"uuid":"c5f9bc37-4bd6-4643-8f0d-e53c5f2c8782","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":3713764498},{"type":"v1","addr":"172.21.15.178:6817","nonce":3713764498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":3713764498},{"type":"v1","addr":"172.21.15.178:6819","nonce":3713764498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":3713764498},{"type":"v1","addr":"172.21.15.178:6823","nonce":3713764498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":3713764498},{"type":"v1","addr":"172.21.15.178:6821","nonce":3713764498}]},"public_addr":"172.21.15.178:6817/3713764498","cluster_addr":"172.21.15.178:6819/3713764498","heartbeat_back_addr":"172.21.15.178:6823/3713764498","heartbeat_front_addr":"172.21.15.178:6821/3713764498","state":["exists","up"]},{"osd":6,"uuid":"d5af075d-3eaa-4cb5-b6e9-6f01c201f2ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1383036222},{"type":"v1","addr":"172.21.15.178:6801","nonce":1383036222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1383036222},{"type":"v1","addr":"172.21.15.178:6803","nonce":1383036222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1383036222},{"type":"v1","addr":"172.21.15.178:6807","nonce":1383036222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1383036222},{"type":"v1","addr":"172.21.15.178:6805","nonce":1383036222}]},"public_addr":"172.21.15.178:6801/1383036222","cluster_addr":"172.21.15.178:6803/1383036222","heartbeat_back_addr":"172.21.15.178:6807/1383036222","heartbeat_front_addr":"172.21.15.178:6805/1383036222","state":["exists","up"]},{"osd":7,"uuid":"323233c7-a3af-4fb0-ac70-de523a279b44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":3049776526},{"type":"v1","addr":"172.21.15.178:6809","nonce":3049776526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":3049776526},{"type":"v1","addr":"172.21.15.178:6811","nonce":3049776526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":3049776526},{"type":"v1","addr":"172.21.15.178:6815","nonce":3049776526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":3049776526},{"type":"v1","addr":"172.21.15.178:6813","nonce":3049776526}]},"public_addr":"172.21.15.178:6809/3049776526","cluster_addr":"172.21.15.178:6811/3049776526","heartbeat_back_addr":"172.21.15.178:6815/3049776526","heartbeat_front_addr":"172.21.15.178:6813/3049776526","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:42:48.855951+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:42:49.166018+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:42:48.891180+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:42:48.243508+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:42:47.750264+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:42:47.225543+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:42:47.826018+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:42:47.675658+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-15T12:43:09.963 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.972+0000 7fe121da1640 10 librados: watch_flush enter 2024-05-15T12:43:09.963 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.972+0000 7fe121da1640 10 librados: watch_flush exit 2024-05-15T12:43:09.963 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.972+0000 7fe121da1640 20 client.4259.objecter shutdown clearing up homeless session... 2024-05-15T12:43:09.963 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.972+0000 7fe121da1640 10 client.4259.objecter successfully canceled tick 2024-05-15T12:43:09.964 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.973+0000 7fe121da1640 1 -- 172.21.15.117:0/2611534735 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fe10806ce90 msgr2=0x7fe10806f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:09.964 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.973+0000 7fe121da1640 1 --2- 172.21.15.117:0/2611534735 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fe10806ce90 0x7fe10806f350 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe10c036000 tx=0x7fe10c03d040 comp rx=0 tx=0).stop 2024-05-15T12:43:09.964 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.973+0000 7fe121da1640 10 monclient: shutdown 2024-05-15T12:43:09.964 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.973+0000 7fe121da1640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:09.964 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.973+0000 7fe121da1640 1 -- 172.21.15.117:0/2611534735 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe11c0b08d0 msgr2=0x7fe11c13cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:09.964 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.973+0000 7fe121da1640 1 --2- 172.21.15.117:0/2611534735 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe11c0b08d0 0x7fe11c13cf00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe11400be20 tx=0x7fe11400bf10 comp rx=0 tx=0).stop 2024-05-15T12:43:09.964 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.973+0000 7fe121da1640 1 -- 172.21.15.117:0/2611534735 shutdown_connections 2024-05-15T12:43:09.964 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.973+0000 7fe121da1640 1 --2- 172.21.15.117:0/2611534735 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe11c0b08d0 0x7fe11c13cf00 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:09.964 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.973+0000 7fe121da1640 1 --2- 172.21.15.117:0/2611534735 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe11c0b4d00 0x7fe11c13c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:09.964 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.973+0000 7fe121da1640 1 --2- 172.21.15.117:0/2611534735 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fe10806ce90 0x7fe10806f350 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:09.964 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.973+0000 7fe121da1640 1 --2- 172.21.15.117:0/2611534735 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe11c0a94b0 0x7fe11c1df730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:09.964 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.973+0000 7fe121da1640 1 -- 172.21.15.117:0/2611534735 >> 172.21.15.117:0/2611534735 conn(0x7fe11c066cc0 msgr2=0x7fe11c074610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:09.964 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.973+0000 7fe121da1640 1 -- 172.21.15.117:0/2611534735 shutdown_connections 2024-05-15T12:43:09.964 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.973+0000 7fe121da1640 1 -- 172.21.15.117:0/2611534735 wait complete. 2024-05-15T12:43:09.965 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:09.973+0000 7fe121da1640 1 librados: shutdown 2024-05-15T12:43:09.974 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-05-15T12:43:09.975 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-15T12:43:10.083 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.093+0000 7f8d46f28640 10 monclient: get_monmap_and_config 2024-05-15T12:43:10.083 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.093+0000 7f8d46f28640 10 monclient: build_initial_monmap 2024-05-15T12:43:10.083 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.093+0000 7f8d46f28640 10 monclient: monmap: 2024-05-15T12:43:10.083 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:10.083 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:10.083 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:43:10.094092+0000 2024-05-15T12:43:10.083 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:43:10.094092+0000 2024-05-15T12:43:10.083 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:10.083 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:10.083 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T12:43:10.083 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T12:43:10.084 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T12:43:10.084 INFO:teuthology.orchestra.run.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T12:43:10.084 INFO:teuthology.orchestra.run.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T12:43:10.084 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:10.086 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.093+0000 7f8d46f28640 1 Processor -- start 2024-05-15T12:43:10.086 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.093+0000 7f8d46f28640 1 -- start start 2024-05-15T12:43:10.086 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.093+0000 7f8d46f28640 10 monclient: init 2024-05-15T12:43:10.086 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.094+0000 7f8d46f28640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:10.086 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.094+0000 7f8d46f28640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-05-15T12:43:10.086 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.094+0000 7f8d46f28640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f8d400b4d00 0x7f8d400b1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:10.086 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.094+0000 7f8d46f28640 10 monclient(hunting): picked mon.noname-a con 0x7f8d400b4d00 addr v2:172.21.15.117:3300/0 2024-05-15T12:43:10.086 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.094+0000 7f8d46f28640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8d400b50e0 addr v1:172.21.15.178:6789/0 2024-05-15T12:43:10.086 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.094+0000 7f8d46f28640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8d400a94b0 0x7f8d400b0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:10.086 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.094+0000 7f8d46f28640 10 monclient(hunting): picked mon.noname-b con 0x7f8d400a94b0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:43:10.087 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.094+0000 7f8d46f28640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d400ab1b0 con 0x7f8d400b50e0 2024-05-15T12:43:10.087 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.094+0000 7f8d46f28640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:10.087 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.094+0000 7f8d46f28640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d400b23c0 con 0x7f8d400b4d00 2024-05-15T12:43:10.087 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.094+0000 7f8d46f28640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:10.087 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.094+0000 7f8d46f28640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d400b1520 con 0x7f8d400a94b0 2024-05-15T12:43:10.087 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.094+0000 7f8d46f28640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:10.087 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.094+0000 7f8d46f28640 10 monclient(hunting): authenticate will time out at 1353.336347s 2024-05-15T12:43:10.087 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.095+0000 7f8d44cc4640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f8d400b4d00 0x7f8d400b1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:10.087 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.095+0000 7f8d454c5640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8d400a94b0 0x7f8d400b0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:10.087 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.095+0000 7f8d3ffff640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f8d400b50e0 0x7f8d40145c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.117:53922/0 (socket says 172.21.15.117:53922) 2024-05-15T12:43:10.087 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.095+0000 7f8d3ffff640 1 -- 172.21.15.117:0/4021592784 learned_addr learned my addr 172.21.15.117:0/4021592784 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-05-15T12:43:10.087 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.095+0000 7f8d44cc4640 10 monclient(hunting): get_auth_request con 0x7f8d400b4d00 auth_method 0 2024-05-15T12:43:10.087 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.095+0000 7f8d44cc4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:10.087 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.095+0000 7f8d44cc4640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:10.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.095+0000 7f8d44cc4640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:10.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.095+0000 7f8d454c5640 10 monclient(hunting): get_auth_request con 0x7f8d400a94b0 auth_method 0 2024-05-15T12:43:10.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.095+0000 7f8d454c5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:10.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.095+0000 7f8d454c5640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:10.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.095+0000 7f8d454c5640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:10.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.096+0000 7f8d44cc4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:10.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.096+0000 7f8d44cc4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:10.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.096+0000 7f8d44cc4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:10.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.096+0000 7f8d454c5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:10.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.096+0000 7f8d454c5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:10.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.096+0000 7f8d454c5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:10.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.097+0000 7f8d3f7fe640 1 -- 172.21.15.117:0/4021592784 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3439642919 0 0) 0x7f8d400ab1b0 con 0x7f8d400b50e0 2024-05-15T12:43:10.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.097+0000 7f8d3f7fe640 10 monclient(hunting): _negotiate 2024-05-15T12:43:10.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.097+0000 7f8d3f7fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:10.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.097+0000 7f8d3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:10.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.097+0000 7f8d3f7fe640 10 monclient(hunting): my global_id is 4306 2024-05-15T12:43:10.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.097+0000 7f8d3f7fe640 1 -- 172.21.15.117:0/4021592784 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d28003520 con 0x7f8d400b50e0 2024-05-15T12:43:10.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.097+0000 7f8d44cc4640 10 monclient(hunting): handle_auth_done global_id 4278 payload 274 2024-05-15T12:43:10.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.097+0000 7f8d44cc4640 1 -- 172.21.15.117:0/4021592784 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8d400a94b0 msgr2=0x7f8d400b0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:43:10.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.097+0000 7f8d44cc4640 1 --2- 172.21.15.117:0/4021592784 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8d400a94b0 0x7f8d400b0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:10.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.097+0000 7f8d44cc4640 1 -- 172.21.15.117:0/4021592784 >> v1:172.21.15.178:6789/0 conn(0x7f8d400b50e0 legacy=0x7f8d40145c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:10.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.097+0000 7f8d44cc4640 10 monclient: _finish_hunting 0 2024-05-15T12:43:10.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.097+0000 7f8d44cc4640 1 monclient: found mon.noname-a 2024-05-15T12:43:10.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.097+0000 7f8d44cc4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:10.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.097+0000 7f8d44cc4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.117:3300/0 2024-05-15T12:43:10.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.097+0000 7f8d44cc4640 1 -- 172.21.15.117:0/4021592784 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d400b16d0 con 0x7f8d400b4d00 2024-05-15T12:43:10.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.100+0000 7f8d44cc4640 1 --2- 172.21.15.117:0/4021592784 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d400b4d00 0x7f8d400b1e80 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8d300097a0 tx=0x7f8d3002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d14fd14e59311362 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:10.094 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.104+0000 7f8d3f7fe640 1 -- 172.21.15.117:0/4021592784 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d300315a0 con 0x7f8d400b4d00 2024-05-15T12:43:10.094 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.104+0000 7f8d3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:10.094 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.104+0000 7f8d3f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T12:43:10.094 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.104+0000 7f8d3f7fe640 10 monclient: dump: 2024-05-15T12:43:10.094 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:10.094 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:10.094 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:10.094 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:10.094 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:10.095 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:10.095 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:10.095 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:10.095 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:10.095 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:10.095 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.104+0000 7f8d3f7fe640 10 monclient: _finish_auth 0 2024-05-15T12:43:10.095 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.104+0000 7f8d3f7fe640 10 monclient: _check_auth_tickets 2024-05-15T12:43:10.095 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.104+0000 7f8d3f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:10.095 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.104+0000 7f8d3f7fe640 1 -- 172.21.15.117:0/4021592784 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d30037070 con 0x7f8d400b4d00 2024-05-15T12:43:10.095 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.104+0000 7f8d46f28640 5 monclient: authenticate success, global_id 4278 2024-05-15T12:43:10.095 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.104+0000 7f8d46f28640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:43:10.095 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.104+0000 7f8d3f7fe640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:10.095 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.104+0000 7f8d3f7fe640 1 -- 172.21.15.117:0/4021592784 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d30031960 con 0x7f8d400b4d00 2024-05-15T12:43:10.095 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.104+0000 7f8d3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:10.095 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.104+0000 7f8d3f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T12:43:10.095 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.104+0000 7f8d3f7fe640 10 monclient: dump: 2024-05-15T12:43:10.096 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:10.096 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:10.096 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:10.096 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:10.096 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:10.096 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:10.096 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:10.096 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:10.096 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:10.096 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:10.096 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.104+0000 7f8d46f28640 10 monclient: get_monmap_and_config success 2024-05-15T12:43:10.096 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.104+0000 7f8d46f28640 4 set_mon_vals no callback set 2024-05-15T12:43:10.100 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.104+0000 7f8d46f28640 10 monclient: shutdown 2024-05-15T12:43:10.100 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.104+0000 7f8d46f28640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:10.100 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.104+0000 7f8d46f28640 1 -- 172.21.15.117:0/4021592784 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d400b4d00 msgr2=0x7f8d400b1e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:10.100 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.104+0000 7f8d46f28640 1 --2- 172.21.15.117:0/4021592784 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d400b4d00 0x7f8d400b1e80 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8d300097a0 tx=0x7f8d3002b840 comp rx=0 tx=0).stop 2024-05-15T12:43:10.100 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.104+0000 7f8d46f28640 1 -- 172.21.15.117:0/4021592784 shutdown_connections 2024-05-15T12:43:10.100 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.104+0000 7f8d46f28640 1 --2- 172.21.15.117:0/4021592784 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8d400a94b0 0x7f8d400b0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:10.100 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.104+0000 7f8d46f28640 1 --2- 172.21.15.117:0/4021592784 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d400b4d00 0x7f8d400b1e80 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:10.100 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.104+0000 7f8d46f28640 1 -- 172.21.15.117:0/4021592784 >> 172.21.15.117:0/4021592784 conn(0x7f8d40066cc0 msgr2=0x7f8d400670d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:10.100 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.104+0000 7f8d46f28640 1 -- 172.21.15.117:0/4021592784 shutdown_connections 2024-05-15T12:43:10.100 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.104+0000 7f8d46f28640 1 -- 172.21.15.117:0/4021592784 wait complete. 2024-05-15T12:43:10.100 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.105+0000 7f8d46f28640 10 monclient: build_initial_monmap 2024-05-15T12:43:10.101 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.105+0000 7f8d46f28640 10 monclient: monmap: 2024-05-15T12:43:10.101 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:10.101 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:43:10.101 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-05-15T12:43:10.101 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-05-15T12:43:10.101 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:10.101 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:10.101 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T12:43:10.101 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T12:43:10.101 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T12:43:10.101 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:10.101 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.105+0000 7f8d46f28640 1 librados: starting msgr at 2024-05-15T12:43:10.101 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.105+0000 7f8d46f28640 1 librados: starting objecter 2024-05-15T12:43:10.101 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.105+0000 7f8d46f28640 1 Processor -- start 2024-05-15T12:43:10.101 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.105+0000 7f8d46f28640 1 -- start start 2024-05-15T12:43:10.101 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.105+0000 7f8d46f28640 1 librados: setting wanted keys 2024-05-15T12:43:10.102 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.105+0000 7f8d46f28640 1 librados: calling monclient init 2024-05-15T12:43:10.102 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.105+0000 7f8d46f28640 10 monclient: init 2024-05-15T12:43:10.102 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.106+0000 7f8d46f28640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:10.102 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.106+0000 7f8d46f28640 10 monclient: _add_conns ranks=[2,1,0] 2024-05-15T12:43:10.102 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.106+0000 7f8d46f28640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8d400a94b0 0x7f8d401daed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:10.102 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.106+0000 7f8d46f28640 10 monclient(hunting): picked mon.noname-c con 0x7f8d400a94b0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:43:10.102 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.106+0000 7f8d46f28640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8d400b4d00 0x7f8d400af290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:10.102 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.106+0000 7f8d46f28640 10 monclient(hunting): picked mon.noname-b con 0x7f8d400b4d00 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:43:10.102 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.106+0000 7f8d46f28640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d401db8f0 0x7f8d400ad540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:10.102 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.106+0000 7f8d46f28640 10 monclient(hunting): picked mon.noname-a con 0x7f8d401db8f0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T12:43:10.102 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.106+0000 7f8d46f28640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:10.102 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.106+0000 7f8d46f28640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4014d130 con 0x7f8d401db8f0 2024-05-15T12:43:10.102 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.106+0000 7f8d46f28640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:10.102 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.106+0000 7f8d46f28640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4014cfb0 con 0x7f8d400b4d00 2024-05-15T12:43:10.103 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.106+0000 7f8d46f28640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:10.103 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.106+0000 7f8d46f28640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4014d2b0 con 0x7f8d400a94b0 2024-05-15T12:43:10.103 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.106+0000 7f8d46f28640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:10.103 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.106+0000 7f8d46f28640 10 monclient(hunting): authenticate will time out at 1353.348459s 2024-05-15T12:43:10.103 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.107+0000 7f8d454c5640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d401db8f0 0x7f8d400ad540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:10.103 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.107+0000 7f8d3ffff640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8d400b4d00 0x7f8d400af290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:10.103 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.107+0000 7f8d44cc4640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8d400a94b0 0x7f8d401daed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:10.103 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.107+0000 7f8d454c5640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d401db8f0 0x7f8d400ad540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:60066/0 (socket says 172.21.15.117:60066) 2024-05-15T12:43:10.103 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.107+0000 7f8d454c5640 1 -- 172.21.15.117:0/2629114411 learned_addr learned my addr 172.21.15.117:0/2629114411 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:43:10.103 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.107+0000 7f8d454c5640 10 monclient(hunting): get_auth_request con 0x7f8d401db8f0 auth_method 0 2024-05-15T12:43:10.103 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.107+0000 7f8d454c5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:10.103 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.107+0000 7f8d454c5640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:10.103 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.107+0000 7f8d454c5640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:10.103 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.107+0000 7f8d44cc4640 10 monclient(hunting): get_auth_request con 0x7f8d400a94b0 auth_method 0 2024-05-15T12:43:10.103 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.107+0000 7f8d44cc4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:10.103 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.107+0000 7f8d44cc4640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:10.104 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.107+0000 7f8d44cc4640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:10.104 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.107+0000 7f8d3ffff640 10 monclient(hunting): get_auth_request con 0x7f8d400b4d00 auth_method 0 2024-05-15T12:43:10.104 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.107+0000 7f8d3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:10.104 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.107+0000 7f8d3ffff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:10.104 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.107+0000 7f8d3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:10.104 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.108+0000 7f8d454c5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:10.104 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.108+0000 7f8d454c5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:10.104 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.108+0000 7f8d454c5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:10.104 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.108+0000 7f8d44cc4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:10.104 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.108+0000 7f8d44cc4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:10.104 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.108+0000 7f8d44cc4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:10.104 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.108+0000 7f8d3ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:10.104 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.108+0000 7f8d3ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:10.104 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.108+0000 7f8d3ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:10.105 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.110+0000 7f8d454c5640 10 monclient(hunting): handle_auth_done global_id 4281 payload 824 2024-05-15T12:43:10.105 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.111+0000 7f8d454c5640 1 -- 172.21.15.117:0/2629114411 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8d400a94b0 msgr2=0x7f8d401daed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:10.105 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.111+0000 7f8d454c5640 1 --2- 172.21.15.117:0/2629114411 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8d400a94b0 0x7f8d401daed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:10.105 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.111+0000 7f8d454c5640 1 -- 172.21.15.117:0/2629114411 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8d400b4d00 msgr2=0x7f8d400af290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:10.105 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.111+0000 7f8d454c5640 1 --2- 172.21.15.117:0/2629114411 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8d400b4d00 0x7f8d400af290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:10.105 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.111+0000 7f8d454c5640 10 monclient: _finish_hunting 0 2024-05-15T12:43:10.106 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.111+0000 7f8d454c5640 1 monclient: found mon.noname-a 2024-05-15T12:43:10.106 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.111+0000 7f8d454c5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:10.106 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.111+0000 7f8d454c5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.117:3300/0 2024-05-15T12:43:10.106 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.111+0000 7f8d454c5640 1 -- 172.21.15.117:0/2629114411 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d400ac810 con 0x7f8d401db8f0 2024-05-15T12:43:10.106 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.113+0000 7f8d454c5640 1 --2- 172.21.15.117:0/2629114411 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d401db8f0 0x7f8d400ad540 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8d3800efc0 tx=0x7f8d3800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:10.106 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.113+0000 7f8d3d7fa640 10 client.?.objecter ms_handle_connect 0x7f8d401db8f0 2024-05-15T12:43:10.106 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.113+0000 7f8d3d7fa640 10 client.?.objecter resend_mon_ops 2024-05-15T12:43:10.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.116+0000 7f8d3d7fa640 1 -- 172.21.15.117:0/2629114411 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d38018020 con 0x7f8d401db8f0 2024-05-15T12:43:10.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.116+0000 7f8d3d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:10.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.116+0000 7f8d3d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T12:43:10.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.116+0000 7f8d3d7fa640 10 monclient: dump: 2024-05-15T12:43:10.107 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:10.107 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:10.107 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:10.107 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:10.108 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:10.108 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:10.108 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:10.108 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:10.108 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:10.108 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:10.108 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.116+0000 7f8d3d7fa640 10 monclient: _finish_auth 0 2024-05-15T12:43:10.108 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.116+0000 7f8d3d7fa640 10 monclient: _check_auth_tickets 2024-05-15T12:43:10.108 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.116+0000 7f8d3d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:10.108 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.116+0000 7f8d3d7fa640 1 -- 172.21.15.117:0/2629114411 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d38009b10 con 0x7f8d401db8f0 2024-05-15T12:43:10.108 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.116+0000 7f8d3d7fa640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:10.108 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.116+0000 7f8d3d7fa640 1 -- 172.21.15.117:0/2629114411 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d380146c0 con 0x7f8d401db8f0 2024-05-15T12:43:10.108 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.116+0000 7f8d46f28640 5 monclient: authenticate success, global_id 4281 2024-05-15T12:43:10.108 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.116+0000 7f8d46f28640 10 monclient: _renew_subs 2024-05-15T12:43:10.108 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.116+0000 7f8d46f28640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:43:10.108 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.116+0000 7f8d46f28640 1 -- 172.21.15.117:0/2629114411 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d400acbd0 con 0x7f8d401db8f0 2024-05-15T12:43:10.108 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.116+0000 7f8d3d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:10.109 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.116+0000 7f8d3d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T12:43:10.109 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.116+0000 7f8d3d7fa640 10 monclient: dump: 2024-05-15T12:43:10.109 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:10.109 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:10.109 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:10.109 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:10.109 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:10.109 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:10.109 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:10.109 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:10.109 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:10.109 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:10.109 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.116+0000 7f8d3effd640 4 set_mon_vals no callback set 2024-05-15T12:43:10.109 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.117+0000 7f8d46f28640 10 client.4281.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:43:10.109 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.117+0000 7f8d46f28640 10 monclient: _renew_subs 2024-05-15T12:43:10.109 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.117+0000 7f8d46f28640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:43:10.109 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.117+0000 7f8d46f28640 1 -- 172.21.15.117:0/2629114411 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d400adc60 con 0x7f8d401db8f0 2024-05-15T12:43:10.110 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.117+0000 7f8d46f28640 1 librados: init done 2024-05-15T12:43:10.110 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.118+0000 7f8d46f28640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:10.110 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.118+0000 7f8d46f28640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:10.110 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.118+0000 7f8d46f28640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:43:10.110 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.118+0000 7f8d46f28640 1 -- 172.21.15.117:0/2629114411 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d4004f0e0 con 0x7f8d401db8f0 2024-05-15T12:43:10.125 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.136+0000 7f8d3d7fa640 1 -- 172.21.15.117:0/2629114411 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f8d38005d10 con 0x7f8d401db8f0 2024-05-15T12:43:10.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.136+0000 7f8d3d7fa640 1 --2- 172.21.15.117:0/2629114411 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f8d2c06cee0 0x7f8d2c06f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:10.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.136+0000 7f8d44cc4640 1 --2- 172.21.15.117:0/2629114411 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f8d2c06cee0 0x7f8d2c06f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:10.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.136+0000 7f8d44cc4640 10 monclient: get_auth_request con 0x7f8d2c06cee0 auth_method 0 2024-05-15T12:43:10.127 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.137+0000 7f8d3d7fa640 10 client.4281.objecter ms_handle_connect 0x7f8d2c06cee0 2024-05-15T12:43:10.127 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.137+0000 7f8d44cc4640 1 --2- 172.21.15.117:0/2629114411 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f8d2c06cee0 0x7f8d2c06f3a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8d30036000 tx=0x7f8d3003d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:10.129 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.139+0000 7f8d3d7fa640 1 -- 172.21.15.117:0/2629114411 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f8d38057c20 con 0x7f8d401db8f0 2024-05-15T12:43:10.129 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.139+0000 7f8d3d7fa640 10 client.4281.objecter ms_dispatch 0x7f8d40000b90 osd_map(25..25 src has 1..25) v4 2024-05-15T12:43:10.129 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.139+0000 7f8d3d7fa640 3 client.4281.objecter handle_osd_map got epochs [25,25] > 0 2024-05-15T12:43:10.129 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.139+0000 7f8d3d7fa640 3 client.4281.objecter handle_osd_map decoding full epoch 25 2024-05-15T12:43:10.130 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.139+0000 7f8d3d7fa640 20 client.4281.objecter dump_active .. 0 homeless 2024-05-15T12:43:10.373 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.383+0000 7f8d3d7fa640 1 -- 172.21.15.117:0/2629114411 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f8d38022030 con 0x7f8d401db8f0 2024-05-15T12:43:10.373 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.383+0000 7f8d3d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:10.373 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.383+0000 7f8d3d7fa640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:43:10.550 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.560+0000 7f8d46f28640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:43:10.550 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.560+0000 7f8d46f28640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:43:10.550 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.560+0000 7f8d46f28640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:43:10.550 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.560+0000 7f8d46f28640 1 -- 172.21.15.117:0/2629114411 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8d400ac9e0 con 0x7f8d401db8f0 2024-05-15T12:43:10.668 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.678+0000 7f8d3d7fa640 1 -- 172.21.15.117:0/2629114411 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+14790 (secure 0 0 0) 0x7f8d38010040 con 0x7f8d401db8f0 2024-05-15T12:43:10.668 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.678+0000 7f8d3d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:43:10.668 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.678+0000 7f8d3d7fa640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:43:10.668 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:43:10.669 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":25,"fsid":"8817f6cf-d933-4825-b972-575ef59f5ba9","created":"2024-05-15T12:42:04.264009+0000","modified":"2024-05-15T12:43:09.352925+0000","last_up_change":"2024-05-15T12:43:01.039161+0000","last_in_change":"2024-05-15T12:42:09.879357+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-15T12:42:52.250376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-05-15T12:43:06.641055+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aaffd6d9-1a00-4793-9d1f-65a30501bec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":1673738625},{"type":"v1","addr":"172.21.15.117:6827","nonce":1673738625}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":1673738625},{"type":"v1","addr":"172.21.15.117:6829","nonce":1673738625}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":1673738625},{"type":"v1","addr":"172.21.15.117:6833","nonce":1673738625}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":1673738625},{"type":"v1","addr":"172.21.15.117:6831","nonce":1673738625}]},"public_addr":"172.21.15.117:6827/1673738625","cluster_addr":"172.21.15.117:6829/1673738625","heartbeat_back_addr":"172.21.15.117:6833/1673738625","heartbeat_front_addr":"172.21.15.117:6831/1673738625","state":["exists","up"]},{"osd":1,"uuid":"efd4e8ed-699c-4ccb-b989-cfadc7aaed35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3864975175},{"type":"v1","addr":"172.21.15.117:6819","nonce":3864975175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3864975175},{"type":"v1","addr":"172.21.15.117:6821","nonce":3864975175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3864975175},{"type":"v1","addr":"172.21.15.117:6825","nonce":3864975175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3864975175},{"type":"v1","addr":"172.21.15.117:6823","nonce":3864975175}]},"public_addr":"172.21.15.117:6819/3864975175","cluster_addr":"172.21.15.117:6821/3864975175","heartbeat_back_addr":"172.21.15.117:6825/3864975175","heartbeat_front_addr":"172.21.15.117:6823/3864975175","state":["exists","up"]},{"osd":2,"uuid":"708e4e3e-7e79-480f-9dbf-7b972c2f2811","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1425835241},{"type":"v1","addr":"172.21.15.117:6805","nonce":1425835241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1425835241},{"type":"v1","addr":"172.21.15.117:6811","nonce":1425835241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":1425835241},{"type":"v1","addr":"172.21.15.117:6817","nonce":1425835241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6813","nonce":1425835241},{"type":"v1","addr":"172.21.15.117:6815","nonce":1425835241}]},"public_addr":"172.21.15.117:6805/1425835241","cluster_addr":"172.21.15.117:6811/1425835241","heartbeat_back_addr":"172.21.15.117:6817/1425835241","heartbeat_front_addr":"172.21.15.117:6815/1425835241","state":["exists","up"]},{"osd":3,"uuid":"e7b5d7d7-515c-42d9-9c41-b1ca3977865e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3842021926},{"type":"v1","addr":"172.21.15.117:6803","nonce":3842021926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3842021926},{"type":"v1","addr":"172.21.15.117:6807","nonce":3842021926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":3842021926},{"type":"v1","addr":"172.21.15.117:6814","nonce":3842021926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":3842021926},{"type":"v1","addr":"172.21.15.117:6809","nonce":3842021926}]},"public_addr":"172.21.15.117:6803/3842021926","cluster_addr":"172.21.15.117:6807/3842021926","heartbeat_back_addr":"172.21.15.117:6814/3842021926","heartbeat_front_addr":"172.21.15.117:6809/3842021926","state":["exists","up"]},{"osd":4,"uuid":"fa647c2c-942b-4be3-afb9-6dba6b0a8cf3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":248472851},{"type":"v1","addr":"172.21.15.178:6825","nonce":248472851}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":248472851},{"type":"v1","addr":"172.21.15.178:6827","nonce":248472851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":248472851},{"type":"v1","addr":"172.21.15.178:6831","nonce":248472851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":248472851},{"type":"v1","addr":"172.21.15.178:6829","nonce":248472851}]},"public_addr":"172.21.15.178:6825/248472851","cluster_addr":"172.21.15.178:6827/248472851","heartbeat_back_addr":"172.21.15.178:6831/248472851","heartbeat_front_addr":"172.21.15.178:6829/248472851","state":["exists","up"]},{"osd":5,"uuid":"c5f9bc37-4bd6-4643-8f0d-e53c5f2c8782","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":3713764498},{"type":"v1","addr":"172.21.15.178:6817","nonce":3713764498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":3713764498},{"type":"v1","addr":"172.21.15.178:6819","nonce":3713764498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":3713764498},{"type":"v1","addr":"172.21.15.178:6823","nonce":3713764498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":3713764498},{"type":"v1","addr":"172.21.15.178:6821","nonce":3713764498}]},"public_addr":"172.21.15.178:6817/3713764498","cluster_addr":"172.21.15.178:6819/3713764498","heartbeat_back_addr":"172.21.15.178:6823/3713764498","heartbeat_front_addr":"172.21.15.178:6821/3713764498","state":["exists","up"]},{"osd":6,"uuid":"d5af075d-3eaa-4cb5-b6e9-6f01c201f2ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1383036222},{"type":"v1","addr":"172.21.15.178:6801","nonce":1383036222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1383036222},{"type":"v1","addr":"172.21.15.178:6803","nonce":1383036222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1383036222},{"type":"v1","addr":"172.21.15.178:6807","nonce":1383036222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1383036222},{"type":"v1","addr":"172.21.15.178:6805","nonce":1383036222}]},"public_addr":"172.21.15.178:6801/1383036222","cluster_addr":"172.21.15.178:6803/1383036222","heartbeat_back_addr":"172.21.15.178:6807/1383036222","heartbeat_front_addr":"172.21.15.178:6805/1383036222","state":["exists","up"]},{"osd":7,"uuid":"323233c7-a3af-4fb0-ac70-de523a279b44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":3049776526},{"type":"v1","addr":"172.21.15.178:6809","nonce":3049776526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":3049776526},{"type":"v1","addr":"172.21.15.178:6811","nonce":3049776526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":3049776526},{"type":"v1","addr":"172.21.15.178:6815","nonce":3049776526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":3049776526},{"type":"v1","addr":"172.21.15.178:6813","nonce":3049776526}]},"public_addr":"172.21.15.178:6809/3049776526","cluster_addr":"172.21.15.178:6811/3049776526","heartbeat_back_addr":"172.21.15.178:6815/3049776526","heartbeat_front_addr":"172.21.15.178:6813/3049776526","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:42:48.855951+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:42:49.166018+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:42:48.891180+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:42:48.243508+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:42:47.750264+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:42:47.225543+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:42:47.826018+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:42:47.675658+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-15T12:43:10.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.680+0000 7f8d46f28640 10 librados: watch_flush enter 2024-05-15T12:43:10.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.680+0000 7f8d46f28640 10 librados: watch_flush exit 2024-05-15T12:43:10.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.680+0000 7f8d46f28640 20 client.4281.objecter shutdown clearing up homeless session... 2024-05-15T12:43:10.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.680+0000 7f8d46f28640 10 client.4281.objecter successfully canceled tick 2024-05-15T12:43:10.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.680+0000 7f8d46f28640 1 -- 172.21.15.117:0/2629114411 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f8d2c06cee0 msgr2=0x7f8d2c06f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:10.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.680+0000 7f8d46f28640 1 --2- 172.21.15.117:0/2629114411 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f8d2c06cee0 0x7f8d2c06f3a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8d30036000 tx=0x7f8d3003d040 comp rx=0 tx=0).stop 2024-05-15T12:43:10.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.680+0000 7f8d46f28640 10 monclient: shutdown 2024-05-15T12:43:10.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.680+0000 7f8d46f28640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:10.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.680+0000 7f8d46f28640 1 -- 172.21.15.117:0/2629114411 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d401db8f0 msgr2=0x7f8d400ad540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:10.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.680+0000 7f8d46f28640 1 --2- 172.21.15.117:0/2629114411 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d401db8f0 0x7f8d400ad540 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8d3800efc0 tx=0x7f8d3800c490 comp rx=0 tx=0).stop 2024-05-15T12:43:10.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.680+0000 7f8d46f28640 1 -- 172.21.15.117:0/2629114411 shutdown_connections 2024-05-15T12:43:10.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.680+0000 7f8d46f28640 1 --2- 172.21.15.117:0/2629114411 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d401db8f0 0x7f8d400ad540 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:10.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.680+0000 7f8d46f28640 1 --2- 172.21.15.117:0/2629114411 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f8d2c06cee0 0x7f8d2c06f3a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:10.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.680+0000 7f8d46f28640 1 --2- 172.21.15.117:0/2629114411 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8d400b4d00 0x7f8d400af290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:10.672 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.680+0000 7f8d46f28640 1 --2- 172.21.15.117:0/2629114411 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8d400a94b0 0x7f8d401daed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:10.672 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.680+0000 7f8d46f28640 1 -- 172.21.15.117:0/2629114411 >> 172.21.15.117:0/2629114411 conn(0x7f8d40066cc0 msgr2=0x7f8d400b65b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:10.672 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.680+0000 7f8d46f28640 1 -- 172.21.15.117:0/2629114411 shutdown_connections 2024-05-15T12:43:10.672 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.680+0000 7f8d46f28640 1 -- 172.21.15.117:0/2629114411 wait complete. 2024-05-15T12:43:10.672 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.680+0000 7f8d46f28640 1 librados: shutdown 2024-05-15T12:43:10.681 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-05-15T12:43:10.682 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-05-15T12:43:10.682 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-05-15T12:43:10.683 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-05-15T12:43:10.683 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-05-15T12:43:10.683 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-05-15T12:43:10.684 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-05-15T12:43:10.684 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-05-15T12:43:10.863 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.866+0000 7f994aed6640 10 monclient: get_monmap_and_config 2024-05-15T12:43:10.863 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.866+0000 7f994aed6640 10 monclient: build_initial_monmap 2024-05-15T12:43:10.863 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.866+0000 7f994aed6640 10 monclient: monmap: 2024-05-15T12:43:10.863 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:10.863 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:10.863 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:43:10.867874+0000 2024-05-15T12:43:10.863 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:43:10.867874+0000 2024-05-15T12:43:10.863 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:10.863 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:10.864 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T12:43:10.864 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T12:43:10.864 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T12:43:10.864 INFO:teuthology.orchestra.run.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T12:43:10.864 INFO:teuthology.orchestra.run.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T12:43:10.864 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:10.864 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.871+0000 7f994aed6640 1 Processor -- start 2024-05-15T12:43:10.870 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.878+0000 7f994aed6640 1 -- start start 2024-05-15T12:43:10.870 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.879+0000 7f801da08640 10 monclient: get_monmap_and_config 2024-05-15T12:43:10.870 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.879+0000 7f801da08640 10 monclient: build_initial_monmap 2024-05-15T12:43:10.870 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.879+0000 7f801da08640 10 monclient: monmap: 2024-05-15T12:43:10.870 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:10.870 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:10.870 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:43:10.880657+0000 2024-05-15T12:43:10.870 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:43:10.880657+0000 2024-05-15T12:43:10.870 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:10.870 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:10.871 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T12:43:10.871 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T12:43:10.871 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T12:43:10.871 INFO:teuthology.orchestra.run.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T12:43:10.871 INFO:teuthology.orchestra.run.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T12:43:10.871 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:10.873 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.878+0000 7f994aed6640 10 monclient: init 2024-05-15T12:43:10.873 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.879+0000 7f994aed6640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:10.873 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.879+0000 7f994aed6640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-05-15T12:43:10.873 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.879+0000 7f994aed6640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f994406bef0 0x7f994414a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:10.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.879+0000 7f994aed6640 10 monclient(hunting): picked mon.noname-c con 0x7f994406bef0 addr v2:172.21.15.178:3300/0 2024-05-15T12:43:10.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.879+0000 7f994aed6640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f994414ac00 0x7f9944071160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:10.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.879+0000 7f994aed6640 10 monclient(hunting): picked mon.noname-a con 0x7f994414ac00 addr v2:172.21.15.117:3300/0 2024-05-15T12:43:10.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.879+0000 7f994aed6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f99440716a0 addr v1:172.21.15.117:6789/0 2024-05-15T12:43:10.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.879+0000 7f994aed6640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9944070780 con 0x7f99440716a0 2024-05-15T12:43:10.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.879+0000 7f994aed6640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:10.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.879+0000 7f994aed6640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f994406e9f0 con 0x7f994414ac00 2024-05-15T12:43:10.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.879+0000 7f994aed6640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:10.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.879+0000 7f994aed6640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9944070a60 con 0x7f994406bef0 2024-05-15T12:43:10.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.879+0000 7f994aed6640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:10.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.880+0000 7f994aed6640 10 monclient(hunting): authenticate will time out at 1354.121804s 2024-05-15T12:43:10.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.880+0000 7f9948c72640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f994406bef0 0x7f994414a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:10.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.881+0000 7f9948c72640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f994406bef0 0x7f994414a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:42336/0 (socket says 172.21.15.117:42336) 2024-05-15T12:43:10.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.881+0000 7f9948c72640 1 -- 172.21.15.117:0/1474945047 learned_addr learned my addr 172.21.15.117:0/1474945047 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:43:10.875 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.881+0000 7f9948c72640 10 monclient(hunting): get_auth_request con 0x7f994406bef0 auth_method 0 2024-05-15T12:43:10.875 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.881+0000 7f9948c72640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:10.875 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.881+0000 7f9948c72640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:10.875 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.881+0000 7f9948c72640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:10.875 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.881+0000 7f9948c72640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:10.875 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.881+0000 7f9948c72640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:10.875 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.881+0000 7f9948c72640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:10.877 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.880+0000 7f801da08640 1 Processor -- start 2024-05-15T12:43:10.877 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.880+0000 7f801da08640 1 -- start start 2024-05-15T12:43:10.877 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.880+0000 7f801da08640 10 monclient: init 2024-05-15T12:43:10.877 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.881+0000 7f801da08640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:10.877 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.881+0000 7f801da08640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-05-15T12:43:10.877 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.881+0000 7f801da08640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f801002e200 0x7f801002e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:10.877 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.881+0000 7f801da08640 10 monclient(hunting): picked mon.noname-b con 0x7f801002e200 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:43:10.877 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.881+0000 7f801da08640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f8010032970 0x7f80100cf580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:10.877 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.881+0000 7f801da08640 10 monclient(hunting): picked mon.noname-a con 0x7f8010032970 addr v2:172.21.15.117:3300/0 2024-05-15T12:43:10.877 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.881+0000 7f801da08640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8010032db0 addr v1:172.21.15.117:6789/0 2024-05-15T12:43:10.877 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.882+0000 7f801da08640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f801002f900 con 0x7f8010032db0 2024-05-15T12:43:10.878 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.882+0000 7f801da08640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:10.878 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.882+0000 7f801da08640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f801002fbe0 con 0x7f8010032970 2024-05-15T12:43:10.878 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.882+0000 7f801da08640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:10.878 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.882+0000 7f801da08640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f801002d720 con 0x7f801002e200 2024-05-15T12:43:10.878 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.882+0000 7f801da08640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:10.878 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.882+0000 7f801da08640 10 monclient(hunting): authenticate will time out at 1354.123845s 2024-05-15T12:43:10.878 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.883+0000 7f9948c72640 10 monclient(hunting): handle_auth_done global_id 4312 payload 274 2024-05-15T12:43:10.878 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.883+0000 7f9948c72640 1 -- 172.21.15.117:0/1474945047 >> v2:172.21.15.117:3300/0 conn(0x7f994414ac00 msgr2=0x7f9944071160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:43:10.878 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.883+0000 7f9948c72640 1 --2- 172.21.15.117:0/1474945047 >> v2:172.21.15.117:3300/0 conn(0x7f994414ac00 0x7f9944071160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:10.878 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.883+0000 7f9948c72640 1 -- 172.21.15.117:0/1474945047 >> v1:172.21.15.117:6789/0 conn(0x7f99440716a0 legacy=0x7f99440af550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:43:10.879 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.883+0000 7f9948c72640 10 monclient: _finish_hunting 0 2024-05-15T12:43:10.879 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.883+0000 7f9948c72640 1 monclient: found mon.noname-c 2024-05-15T12:43:10.879 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.883+0000 7f9948c72640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:10.879 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.883+0000 7f9948c72640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.178:3300/0 2024-05-15T12:43:10.879 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.883+0000 7f9948c72640 1 -- 172.21.15.117:0/1474945047 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9944070d40 con 0x7f994406bef0 2024-05-15T12:43:10.881 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.884+0000 7f6586164640 10 monclient: get_monmap_and_config 2024-05-15T12:43:10.881 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.884+0000 7f6586164640 10 monclient: build_initial_monmap 2024-05-15T12:43:10.881 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.884+0000 7f6586164640 10 monclient: monmap: 2024-05-15T12:43:10.881 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:10.881 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:10.881 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:43:10.885731+0000 2024-05-15T12:43:10.882 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:43:10.885731+0000 2024-05-15T12:43:10.882 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:10.882 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:10.882 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T12:43:10.882 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T12:43:10.882 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T12:43:10.882 INFO:teuthology.orchestra.run.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T12:43:10.882 INFO:teuthology.orchestra.run.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T12:43:10.882 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:10.882 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.885+0000 7f6586164640 1 Processor -- start 2024-05-15T12:43:10.882 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.886+0000 7f6586164640 1 -- start start 2024-05-15T12:43:10.882 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.886+0000 7f6586164640 10 monclient: init 2024-05-15T12:43:10.882 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.887+0000 7f6586164640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:10.882 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.887+0000 7f6586164640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-05-15T12:43:10.882 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.887+0000 7f6586164640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6578007200 0x7f6578030860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:10.883 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.887+0000 7f6586164640 10 monclient(hunting): picked mon.noname-b con 0x7f6578007200 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:43:10.883 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.887+0000 7f6586164640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f65780cfae0 0x7f65780cff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:10.883 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.887+0000 7f6586164640 10 monclient(hunting): picked mon.noname-a con 0x7f65780cfae0 addr v2:172.21.15.117:3300/0 2024-05-15T12:43:10.883 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.887+0000 7f6586164640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f6578033590 0x7f6578033a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:10.883 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.887+0000 7f6586164640 10 monclient(hunting): picked mon.noname-c con 0x7f6578033590 addr v2:172.21.15.178:3300/0 2024-05-15T12:43:10.883 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.887+0000 7f6586164640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:10.883 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.887+0000 7f6586164640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65780d0460 con 0x7f65780cfae0 2024-05-15T12:43:10.883 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.887+0000 7f6586164640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:10.883 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.887+0000 7f6586164640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6578037620 con 0x7f6578033590 2024-05-15T12:43:10.883 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.887+0000 7f6586164640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:10.883 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.887+0000 7f6586164640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65780cf5c0 con 0x7f6578007200 2024-05-15T12:43:10.883 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.887+0000 7f6586164640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:10.883 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.887+0000 7f6586164640 10 monclient(hunting): authenticate will time out at 1354.129024s 2024-05-15T12:43:10.884 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.891+0000 7f657ffff640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f6578033590 0x7f6578033a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:10.884 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.891+0000 7f657ffff640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f6578033590 0x7f6578033a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:42340/0 (socket says 172.21.15.117:42340) 2024-05-15T12:43:10.884 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.891+0000 7f657ffff640 1 -- 172.21.15.117:0/3008977338 learned_addr learned my addr 172.21.15.117:0/3008977338 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:43:10.884 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.891+0000 7f657ffff640 10 monclient(hunting): get_auth_request con 0x7f6578033590 auth_method 0 2024-05-15T12:43:10.884 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.891+0000 7f657ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:10.884 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.891+0000 7f657ffff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:10.884 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.891+0000 7f657ffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:10.884 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.891+0000 7f801d207640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f8010032db0 0x7f80100cc080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:48656/0 (socket says 172.21.15.117:48656) 2024-05-15T12:43:10.884 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.891+0000 7f8017fff640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f8010032970 0x7f80100cf580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:10.884 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.891+0000 7f8017fff640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f8010032970 0x7f80100cf580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:60080/0 (socket says 172.21.15.117:60080) 2024-05-15T12:43:10.884 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.891+0000 7f801d207640 1 -- 172.21.15.117:0/284341976 learned_addr learned my addr 172.21.15.117:0/284341976 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-05-15T12:43:10.884 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.891+0000 7f8017fff640 10 monclient(hunting): get_auth_request con 0x7f8010032970 auth_method 0 2024-05-15T12:43:10.885 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.891+0000 7f8017fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:10.885 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.891+0000 7f8017fff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:10.885 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.891+0000 7f8017fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:10.885 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.891+0000 7f657ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:10.885 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.891+0000 7f657ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:10.885 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.891+0000 7f657ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:10.885 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.891+0000 7f9948c72640 1 --2- 172.21.15.117:0/1474945047 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f994406bef0 0x7f994414a6c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f99440672c0 tx=0x7f993c02bdc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=641b52e06b5043ad server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:10.886 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.892+0000 7f6d4c9d3640 10 monclient: get_monmap_and_config 2024-05-15T12:43:10.886 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.892+0000 7f6d4c9d3640 10 monclient: build_initial_monmap 2024-05-15T12:43:10.886 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.892+0000 7f6d4c9d3640 10 monclient: monmap: 2024-05-15T12:43:10.886 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:10.886 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:10.886 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:43:10.893108+0000 2024-05-15T12:43:10.886 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:43:10.893108+0000 2024-05-15T12:43:10.886 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:10.886 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:10.887 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T12:43:10.887 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T12:43:10.887 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T12:43:10.887 INFO:teuthology.orchestra.run.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T12:43:10.887 INFO:teuthology.orchestra.run.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T12:43:10.887 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:10.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.893+0000 7f657ffff640 10 monclient(hunting): handle_auth_done global_id 4315 payload 274 2024-05-15T12:43:10.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.893+0000 7f657ffff640 1 -- 172.21.15.117:0/3008977338 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6578007200 msgr2=0x7f6578030860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:43:10.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.893+0000 7f657ffff640 1 --2- 172.21.15.117:0/3008977338 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6578007200 0x7f6578030860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:10.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.893+0000 7f657ffff640 1 -- 172.21.15.117:0/3008977338 >> v2:172.21.15.117:3300/0 conn(0x7f65780cfae0 msgr2=0x7f65780cff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:43:10.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.893+0000 7f657ffff640 1 --2- 172.21.15.117:0/3008977338 >> v2:172.21.15.117:3300/0 conn(0x7f65780cfae0 0x7f65780cff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:10.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.893+0000 7f657ffff640 10 monclient: _finish_hunting 0 2024-05-15T12:43:10.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.893+0000 7f657ffff640 1 monclient: found mon.noname-c 2024-05-15T12:43:10.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.893+0000 7f657ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:10.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.893+0000 7f657ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.178:3300/0 2024-05-15T12:43:10.889 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.893+0000 7f657ffff640 1 -- 172.21.15.117:0/3008977338 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6578033f60 con 0x7f6578033590 2024-05-15T12:43:10.889 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.893+0000 7f6d4c9d3640 1 Processor -- start 2024-05-15T12:43:10.889 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.893+0000 7f6d4c9d3640 1 -- start start 2024-05-15T12:43:10.889 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.893+0000 7f6d4c9d3640 10 monclient: init 2024-05-15T12:43:10.889 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.894+0000 7f6d4c9d3640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:10.890 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.894+0000 7f6d4c9d3640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-05-15T12:43:10.890 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.894+0000 7f6d4c9d3640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d400071d0 0x7f6d40030830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:10.890 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.894+0000 7f6d4c9d3640 10 monclient(hunting): picked mon.noname-b con 0x7f6d400071d0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:43:10.890 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.894+0000 7f6d4c9d3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6d400cfab0 addr v1:172.21.15.117:6789/0 2024-05-15T12:43:10.890 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.894+0000 7f6d4c9d3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6d40033560 addr v1:172.21.15.178:6789/0 2024-05-15T12:43:10.890 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.894+0000 7f6d4c9d3640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d400375f0 con 0x7f6d400cfab0 2024-05-15T12:43:10.890 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.894+0000 7f6d4c9d3640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d400cf590 con 0x7f6d40033560 2024-05-15T12:43:10.890 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.894+0000 7f6d4c9d3640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:10.890 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.894+0000 7f6d4c9d3640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d40035130 con 0x7f6d400071d0 2024-05-15T12:43:10.890 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.894+0000 7f6d4c9d3640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:10.890 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.894+0000 7f6d4c9d3640 10 monclient(hunting): authenticate will time out at 1354.136571s 2024-05-15T12:43:10.891 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.898+0000 7f99437fe640 1 -- 172.21.15.117:0/1474945047 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993c02c960 con 0x7f994406bef0 2024-05-15T12:43:10.891 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.898+0000 7f99437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:10.891 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.898+0000 7f99437fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:43:10.891 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.898+0000 7f99437fe640 10 monclient: dump: 2024-05-15T12:43:10.891 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:10.891 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:10.891 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:10.891 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:10.892 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:10.892 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:10.892 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:10.892 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:10.892 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:10.892 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:10.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.898+0000 7f99437fe640 10 monclient: _finish_auth 0 2024-05-15T12:43:10.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.898+0000 7f99437fe640 10 monclient: _check_auth_tickets 2024-05-15T12:43:10.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.898+0000 7f99437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:10.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.898+0000 7f99437fe640 1 -- 172.21.15.117:0/1474945047 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f993c02cb00 con 0x7f994406bef0 2024-05-15T12:43:10.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.898+0000 7f99437fe640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:10.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.898+0000 7f99437fe640 1 -- 172.21.15.117:0/1474945047 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993c0343f0 con 0x7f994406bef0 2024-05-15T12:43:10.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.898+0000 7f99437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:10.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.898+0000 7f99437fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:43:10.893 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.898+0000 7f99437fe640 10 monclient: dump: 2024-05-15T12:43:10.893 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:10.893 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:10.893 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:10.893 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:10.893 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:10.893 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:10.893 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:10.893 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:10.893 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:10.893 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:10.893 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.898+0000 7f994aed6640 5 monclient: authenticate success, global_id 4312 2024-05-15T12:43:10.893 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.898+0000 7f994aed6640 10 monclient: get_monmap_and_config success 2024-05-15T12:43:10.893 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.898+0000 7f994aed6640 4 set_mon_vals no callback set 2024-05-15T12:43:10.893 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.899+0000 7f994aed6640 10 monclient: shutdown 2024-05-15T12:43:10.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.899+0000 7f994aed6640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:10.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.899+0000 7f994aed6640 1 -- 172.21.15.117:0/1474945047 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f994406bef0 msgr2=0x7f994414a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:10.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.899+0000 7f994aed6640 1 --2- 172.21.15.117:0/1474945047 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f994406bef0 0x7f994414a6c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f99440672c0 tx=0x7f993c02bdc0 comp rx=0 tx=0).stop 2024-05-15T12:43:10.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.899+0000 7f657ffff640 1 --2- 172.21.15.117:0/3008977338 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6578033590 0x7f6578033a20 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f656c009870 tx=0x7f656c02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=81d7951dd9a183c1 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:10.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.899+0000 7f6d47fff640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f6d40033560 0x7f6d40033a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.117:53936/0 (socket says 172.21.15.117:53936) 2024-05-15T12:43:10.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.899+0000 7f6d47fff640 1 -- 172.21.15.117:0/4092359090 learned_addr learned my addr 172.21.15.117:0/4092359090 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-05-15T12:43:10.896 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.899+0000 7f994aed6640 1 -- 172.21.15.117:0/1474945047 shutdown_connections 2024-05-15T12:43:10.896 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.899+0000 7f994aed6640 1 --2- 172.21.15.117:0/1474945047 >> v2:172.21.15.117:3300/0 conn(0x7f994414ac00 0x7f9944071160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:10.896 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.899+0000 7f994aed6640 1 --2- 172.21.15.117:0/1474945047 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f994406bef0 0x7f994414a6c0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:10.896 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.899+0000 7f994aed6640 1 -- 172.21.15.117:0/1474945047 >> 172.21.15.117:0/1474945047 conn(0x7f994406ada0 msgr2=0x7f994406b1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:10.896 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.899+0000 7f994aed6640 1 -- 172.21.15.117:0/1474945047 shutdown_connections 2024-05-15T12:43:10.896 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.899+0000 7f994aed6640 1 -- 172.21.15.117:0/1474945047 wait complete. 2024-05-15T12:43:10.896 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.900+0000 7f994aed6640 10 monclient: build_initial_monmap 2024-05-15T12:43:10.896 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.900+0000 7f994aed6640 10 monclient: monmap: 2024-05-15T12:43:10.896 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:10.896 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:43:10.896 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-05-15T12:43:10.896 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-05-15T12:43:10.896 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:10.897 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:10.897 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T12:43:10.897 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T12:43:10.897 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T12:43:10.897 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:10.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.900+0000 7f994aed6640 1 librados: starting msgr at 2024-05-15T12:43:10.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.901+0000 7f994aed6640 1 librados: starting objecter 2024-05-15T12:43:10.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.901+0000 7f994aed6640 1 Processor -- start 2024-05-15T12:43:10.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.901+0000 7f994aed6640 1 -- start start 2024-05-15T12:43:10.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.901+0000 7f994aed6640 1 librados: setting wanted keys 2024-05-15T12:43:10.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.901+0000 7f994aed6640 1 librados: calling monclient init 2024-05-15T12:43:10.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.901+0000 7f994aed6640 10 monclient: init 2024-05-15T12:43:10.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.902+0000 7f994aed6640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:10.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.902+0000 7f994aed6640 10 monclient: _add_conns ranks=[2,1,0] 2024-05-15T12:43:10.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.902+0000 7f994aed6640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f994406bef0 0x7f994414b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:10.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.902+0000 7f994aed6640 10 monclient(hunting): picked mon.noname-c con 0x7f994406bef0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:43:10.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.902+0000 7f994aed6640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f99440716a0 0x7f994414d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:10.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.902+0000 7f994aed6640 10 monclient(hunting): picked mon.noname-b con 0x7f99440716a0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:43:10.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.902+0000 7f994aed6640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f994414ac00 0x7f9944149280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:10.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.902+0000 7f994aed6640 10 monclient(hunting): picked mon.noname-a con 0x7f994414ac00 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T12:43:10.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.902+0000 7f9e267d8640 10 monclient: get_monmap_and_config 2024-05-15T12:43:10.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.902+0000 7f9e267d8640 10 monclient: build_initial_monmap 2024-05-15T12:43:10.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.902+0000 7f9e267d8640 10 monclient: monmap: 2024-05-15T12:43:10.899 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:10.899 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:10.899 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:43:10.903520+0000 2024-05-15T12:43:10.899 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:43:10.903520+0000 2024-05-15T12:43:10.899 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:10.899 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:10.899 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T12:43:10.899 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T12:43:10.899 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T12:43:10.899 INFO:teuthology.orchestra.run.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T12:43:10.899 INFO:teuthology.orchestra.run.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T12:43:10.899 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:10.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.902+0000 7f994aed6640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:10.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.902+0000 7f994aed6640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994406fa00 con 0x7f994414ac00 2024-05-15T12:43:10.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.902+0000 7f994aed6640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:10.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.902+0000 7f994aed6640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994414b030 con 0x7f99440716a0 2024-05-15T12:43:10.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.902+0000 7f994aed6640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:10.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.902+0000 7f994aed6640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f994406ee90 con 0x7f994406bef0 2024-05-15T12:43:10.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.902+0000 7f994aed6640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:10.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.902+0000 7f994aed6640 10 monclient(hunting): authenticate will time out at 1354.144493s 2024-05-15T12:43:10.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.903+0000 7f657e7fc640 1 -- 172.21.15.117:0/3008977338 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f656c034070 con 0x7f6578033590 2024-05-15T12:43:10.902 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.903+0000 7f657e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:10.902 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.903+0000 7f657e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:43:10.902 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.903+0000 7f657e7fc640 10 monclient: dump: 2024-05-15T12:43:10.902 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:10.902 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:10.902 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:10.902 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:10.902 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:10.902 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:10.902 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:10.902 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:10.902 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:10.902 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:10.902 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.903+0000 7f657e7fc640 10 monclient: _finish_auth 0 2024-05-15T12:43:10.903 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.903+0000 7f657e7fc640 10 monclient: _check_auth_tickets 2024-05-15T12:43:10.903 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.903+0000 7f657e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:10.903 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.903+0000 7f6586164640 5 monclient: authenticate success, global_id 4315 2024-05-15T12:43:10.903 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.903+0000 7f6586164640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:43:10.903 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.903+0000 7f9e267d8640 1 Processor -- start 2024-05-15T12:43:10.903 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.903+0000 7f9e267d8640 1 -- start start 2024-05-15T12:43:10.903 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.903+0000 7f9e267d8640 10 monclient: init 2024-05-15T12:43:10.904 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.904+0000 7f09266ef640 10 monclient: get_monmap_and_config 2024-05-15T12:43:10.904 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.904+0000 7f09266ef640 10 monclient: build_initial_monmap 2024-05-15T12:43:10.904 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.904+0000 7f09266ef640 10 monclient: monmap: 2024-05-15T12:43:10.904 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:10.904 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:10.904 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:43:10.905288+0000 2024-05-15T12:43:10.904 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:43:10.905288+0000 2024-05-15T12:43:10.904 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:10.904 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:10.904 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T12:43:10.904 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T12:43:10.904 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T12:43:10.904 INFO:teuthology.orchestra.run.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T12:43:10.904 INFO:teuthology.orchestra.run.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T12:43:10.904 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:10.905 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.904+0000 7f9943fff640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f99440716a0 0x7f994414d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:10.905 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.904+0000 7f9e267d8640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:10.905 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.904+0000 7f9e267d8640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-05-15T12:43:10.905 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.904+0000 7f9e267d8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9e20066ae0 addr v1:172.21.15.117:6789/0 2024-05-15T12:43:10.905 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.904+0000 7f9e267d8640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9e20066ec0 0x7f9e20145b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:10.905 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.904+0000 7f9e267d8640 10 monclient(hunting): picked mon.noname-b con 0x7f9e20066ec0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:43:10.905 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.904+0000 7f9e267d8640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f9e200adce0 0x7f9e20146060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:10.906 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.904+0000 7f9e267d8640 10 monclient(hunting): picked mon.noname-a con 0x7f9e200adce0 addr v2:172.21.15.117:3300/0 2024-05-15T12:43:10.906 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.904+0000 7f9e267d8640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e2013ca00 con 0x7f9e20066ae0 2024-05-15T12:43:10.906 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.904+0000 7f9e267d8640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:10.906 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.904+0000 7f9e267d8640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e2006fb00 con 0x7f9e200adce0 2024-05-15T12:43:10.906 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.904+0000 7f9e267d8640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:10.906 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.904+0000 7f9e267d8640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2006fde0 con 0x7f9e20066ec0 2024-05-15T12:43:10.906 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.904+0000 7f9e267d8640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:10.906 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.904+0000 7f9e267d8640 10 monclient(hunting): authenticate will time out at 1354.146402s 2024-05-15T12:43:10.907 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.904+0000 7f9943fff640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f99440716a0 0x7f994414d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:42342/0 (socket says 172.21.15.117:42342) 2024-05-15T12:43:10.907 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.904+0000 7f9943fff640 1 -- 172.21.15.117:0/363281595 learned_addr learned my addr 172.21.15.117:0/363281595 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:43:10.907 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.904+0000 7f9943fff640 10 monclient(hunting): get_auth_request con 0x7f99440716a0 auth_method 0 2024-05-15T12:43:10.907 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.904+0000 7f9943fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:10.907 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.904+0000 7f9943fff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:10.907 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.904+0000 7f9943fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:10.907 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.905+0000 7f09266ef640 1 Processor -- start 2024-05-15T12:43:10.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.905+0000 7f09266ef640 1 -- start start 2024-05-15T12:43:10.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.905+0000 7f09266ef640 10 monclient: init 2024-05-15T12:43:10.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.905+0000 7f9943fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:10.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.905+0000 7f9943fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:10.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.905+0000 7f9943fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:10.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.906+0000 7f09266ef640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:10.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.906+0000 7f09266ef640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-05-15T12:43:10.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.906+0000 7f09266ef640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f09200b4d00 0x7f09200b1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:10.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.906+0000 7f09266ef640 10 monclient(hunting): picked mon.noname-c con 0x7f09200b4d00 addr v2:172.21.15.178:3300/0 2024-05-15T12:43:10.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.906+0000 7f09266ef640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f09200b50e0 0x7f0920145c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:10.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.906+0000 7f09266ef640 10 monclient(hunting): picked mon.noname-a con 0x7f09200b50e0 addr v2:172.21.15.117:3300/0 2024-05-15T12:43:10.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.906+0000 7f09266ef640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f09200a94b0 addr v1:172.21.15.117:6789/0 2024-05-15T12:43:10.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.906+0000 7f09266ef640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09200ab1b0 con 0x7f09200a94b0 2024-05-15T12:43:10.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.906+0000 7f09266ef640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:10.910 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.906+0000 7f09266ef640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09200b23c0 con 0x7f09200b50e0 2024-05-15T12:43:10.910 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.906+0000 7f09266ef640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:10.910 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.906+0000 7f09266ef640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09200b16f0 con 0x7f09200b4d00 2024-05-15T12:43:10.910 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.906+0000 7f09266ef640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:10.910 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.906+0000 7f09266ef640 10 monclient(hunting): authenticate will time out at 1354.148104s 2024-05-15T12:43:10.911 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.910+0000 7f9943fff640 10 monclient(hunting): handle_auth_done global_id 4318 payload 824 2024-05-15T12:43:10.911 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.910+0000 7f9943fff640 1 -- 172.21.15.117:0/363281595 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f994406bef0 msgr2=0x7f994414b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:10.911 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.910+0000 7f9943fff640 1 --2- 172.21.15.117:0/363281595 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f994406bef0 0x7f994414b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:10.911 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.910+0000 7f9943fff640 1 -- 172.21.15.117:0/363281595 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f994414ac00 msgr2=0x7f9944149280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:10.911 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.910+0000 7f9943fff640 1 --2- 172.21.15.117:0/363281595 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f994414ac00 0x7f9944149280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:10.911 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.910+0000 7f9943fff640 10 monclient: _finish_hunting 0 2024-05-15T12:43:10.911 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.910+0000 7f9943fff640 1 monclient: found mon.noname-b 2024-05-15T12:43:10.911 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.910+0000 7f9943fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:10.912 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.910+0000 7f9943fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2024-05-15T12:43:10.912 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.910+0000 7f9943fff640 1 -- 172.21.15.117:0/363281595 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f994414ccd0 con 0x7f99440716a0 2024-05-15T12:43:10.912 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.913+0000 7f657e7fc640 1 -- 172.21.15.117:0/3008977338 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f656c030840 con 0x7f6578033590 2024-05-15T12:43:10.912 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.913+0000 7f657e7fc640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:10.912 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.913+0000 7f657e7fc640 1 -- 172.21.15.117:0/3008977338 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f656c030c10 con 0x7f6578033590 2024-05-15T12:43:10.912 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.913+0000 7f657e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:10.912 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.913+0000 7f657e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:43:10.912 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.913+0000 7f657e7fc640 10 monclient: dump: 2024-05-15T12:43:10.912 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:10.912 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:10.912 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:10.912 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:10.912 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:10.913 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:10.913 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:10.913 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:10.913 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:10.913 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:10.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.913+0000 7f6586164640 10 monclient: get_monmap_and_config success 2024-05-15T12:43:10.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.913+0000 7f6586164640 4 set_mon_vals no callback set 2024-05-15T12:43:10.914 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.913+0000 7f3847cbe640 10 monclient: get_monmap_and_config 2024-05-15T12:43:10.914 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.913+0000 7f3847cbe640 10 monclient: build_initial_monmap 2024-05-15T12:43:10.914 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.913+0000 7f3847cbe640 10 monclient: monmap: 2024-05-15T12:43:10.914 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:10.914 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:10.914 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:43:10.914703+0000 2024-05-15T12:43:10.914 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:43:10.914703+0000 2024-05-15T12:43:10.914 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:10.914 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:10.914 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T12:43:10.914 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T12:43:10.914 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T12:43:10.915 INFO:teuthology.orchestra.run.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T12:43:10.915 INFO:teuthology.orchestra.run.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T12:43:10.915 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:10.915 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.914+0000 7f3847cbe640 1 Processor -- start 2024-05-15T12:43:10.915 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.914+0000 7f3847cbe640 1 -- start start 2024-05-15T12:43:10.915 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.914+0000 7f3847cbe640 10 monclient: init 2024-05-15T12:43:10.915 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.914+0000 7f6586164640 10 monclient: shutdown 2024-05-15T12:43:10.915 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.914+0000 7f6586164640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:10.915 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.914+0000 7f6586164640 1 -- 172.21.15.117:0/3008977338 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6578033590 msgr2=0x7f6578033a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:10.915 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.914+0000 7f6586164640 1 --2- 172.21.15.117:0/3008977338 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6578033590 0x7f6578033a20 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f656c009870 tx=0x7f656c02b980 comp rx=0 tx=0).stop 2024-05-15T12:43:10.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.914+0000 7f091ffff640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f09200b4d00 0x7f09200b1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:10.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.914+0000 7f6586164640 1 -- 172.21.15.117:0/3008977338 shutdown_connections 2024-05-15T12:43:10.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.914+0000 7f6586164640 1 --2- 172.21.15.117:0/3008977338 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6578033590 0x7f6578033a20 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:10.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.914+0000 7f6586164640 1 --2- 172.21.15.117:0/3008977338 >> v2:172.21.15.117:3300/0 conn(0x7f65780cfae0 0x7f65780cff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:10.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.914+0000 7f6586164640 1 --2- 172.21.15.117:0/3008977338 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6578007200 0x7f6578030860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:10.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.914+0000 7f6586164640 1 -- 172.21.15.117:0/3008977338 >> 172.21.15.117:0/3008977338 conn(0x7f65780061d0 msgr2=0x7f65780065e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:10.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.914+0000 7f6586164640 1 -- 172.21.15.117:0/3008977338 shutdown_connections 2024-05-15T12:43:10.917 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.915+0000 7f091ffff640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f09200b4d00 0x7f09200b1e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:42358/0 (socket says 172.21.15.117:42358) 2024-05-15T12:43:10.917 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.915+0000 7f091ffff640 1 -- 172.21.15.117:0/2594824507 learned_addr learned my addr 172.21.15.117:0/2594824507 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:43:10.917 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.915+0000 7f091ffff640 10 monclient(hunting): get_auth_request con 0x7f09200b4d00 auth_method 0 2024-05-15T12:43:10.917 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.915+0000 7f091ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:10.917 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.915+0000 7f091ffff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:10.917 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.915+0000 7f091ffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:10.917 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.914+0000 7f6586164640 1 -- 172.21.15.117:0/3008977338 wait complete. 2024-05-15T12:43:10.918 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.915+0000 7f3847cbe640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:10.918 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.915+0000 7f3847cbe640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-05-15T12:43:10.918 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.915+0000 7f3847cbe640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f38400b4d00 0x7f38400b1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:10.919 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.915+0000 7f3847cbe640 10 monclient(hunting): picked mon.noname-c con 0x7f38400b4d00 addr v2:172.21.15.178:3300/0 2024-05-15T12:43:10.919 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.915+0000 7f3847cbe640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f38400b50e0 0x7f3840145c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:10.919 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.915+0000 7f3847cbe640 10 monclient(hunting): picked mon.noname-b con 0x7f38400b50e0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:43:10.919 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.915+0000 7f6586164640 10 monclient: build_initial_monmap 2024-05-15T12:43:10.919 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.915+0000 7f6586164640 10 monclient: monmap: 2024-05-15T12:43:10.919 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:10.919 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:43:10.919 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-05-15T12:43:10.919 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-05-15T12:43:10.919 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:10.919 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:10.920 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T12:43:10.920 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T12:43:10.920 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T12:43:10.920 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:10.921 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.915+0000 7f3847cbe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f38400a94b0 addr v1:172.21.15.178:6789/0 2024-05-15T12:43:10.921 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.915+0000 7f3847cbe640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38400ab1b0 con 0x7f38400a94b0 2024-05-15T12:43:10.921 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.915+0000 7f3847cbe640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:10.921 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.915+0000 7f3847cbe640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38400b23c0 con 0x7f38400b4d00 2024-05-15T12:43:10.921 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.915+0000 7f3847cbe640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:10.921 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.915+0000 7f3847cbe640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38400b16f0 con 0x7f38400b50e0 2024-05-15T12:43:10.921 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.915+0000 7f3847cbe640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:10.921 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.915+0000 7f3847cbe640 10 monclient(hunting): authenticate will time out at 1354.157722s 2024-05-15T12:43:10.921 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.915+0000 7f091ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:10.922 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.915+0000 7f091ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:10.922 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.915+0000 7f091ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:10.923 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.916+0000 7f6586164640 1 librados: starting msgr at 2024-05-15T12:43:10.923 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.916+0000 7f6586164640 1 librados: starting objecter 2024-05-15T12:43:10.923 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.916+0000 7f6586164640 1 Processor -- start 2024-05-15T12:43:10.923 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.916+0000 7f6586164640 1 -- start start 2024-05-15T12:43:10.923 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.916+0000 7f6586164640 1 librados: setting wanted keys 2024-05-15T12:43:10.923 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.916+0000 7f6586164640 1 librados: calling monclient init 2024-05-15T12:43:10.923 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.916+0000 7f6586164640 10 monclient: init 2024-05-15T12:43:10.923 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.916+0000 7f6586164640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:10.924 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.916+0000 7f6586164640 10 monclient: _add_conns ranks=[0,2,1] 2024-05-15T12:43:10.924 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.916+0000 7f6586164640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6578007200 0x7f6578164b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:10.924 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.917+0000 7f6586164640 10 monclient(hunting): picked mon.noname-a con 0x7f6578007200 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T12:43:10.924 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.917+0000 7f6586164640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6578033590 0x7f657802eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:10.924 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.917+0000 7f6586164640 10 monclient(hunting): picked mon.noname-c con 0x7f6578033590 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:43:10.924 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.917+0000 7f6586164640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f65780cfae0 0x7f65780c5b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:10.924 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.917+0000 7f6586164640 10 monclient(hunting): picked mon.noname-b con 0x7f65780cfae0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:43:10.924 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.917+0000 7f6586164640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:10.924 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.917+0000 7f6586164640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6578038f60 con 0x7f6578007200 2024-05-15T12:43:10.924 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.917+0000 7f6586164640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:10.924 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.917+0000 7f6586164640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6578038de0 con 0x7f65780cfae0 2024-05-15T12:43:10.924 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.917+0000 7f6586164640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:10.924 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.917+0000 7f6586164640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65780359b0 con 0x7f6578033590 2024-05-15T12:43:10.925 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.917+0000 7f6586164640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:10.925 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.917+0000 7f6586164640 10 monclient(hunting): authenticate will time out at 1354.158859s 2024-05-15T12:43:10.925 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.917+0000 7f8017fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:10.925 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.917+0000 7f8017fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:10.925 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.917+0000 7f8017fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:10.926 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.917+0000 7f091ffff640 10 monclient(hunting): handle_auth_done global_id 4321 payload 274 2024-05-15T12:43:10.926 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.917+0000 7f091ffff640 1 -- 172.21.15.117:0/2594824507 >> v2:172.21.15.117:3300/0 conn(0x7f09200b50e0 msgr2=0x7f0920145c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:43:10.926 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.917+0000 7f091ffff640 1 --2- 172.21.15.117:0/2594824507 >> v2:172.21.15.117:3300/0 conn(0x7f09200b50e0 0x7f0920145c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:10.926 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.917+0000 7f091ffff640 1 -- 172.21.15.117:0/2594824507 >> v1:172.21.15.117:6789/0 conn(0x7f09200a94b0 legacy=0x7f09200b0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:43:10.926 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.918+0000 7f091ffff640 10 monclient: _finish_hunting 0 2024-05-15T12:43:10.926 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.918+0000 7f091ffff640 1 monclient: found mon.noname-c 2024-05-15T12:43:10.926 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.918+0000 7f091ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:10.926 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.918+0000 7f091ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.178:3300/0 2024-05-15T12:43:10.926 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.918+0000 7f091ffff640 1 -- 172.21.15.117:0/2594824507 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0920146160 con 0x7f09200b4d00 2024-05-15T12:43:10.927 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.918+0000 7f3845a5a640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f38400b4d00 0x7f38400b1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:10.927 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.919+0000 7f3845a5a640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f38400b4d00 0x7f38400b1e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:42366/0 (socket says 172.21.15.117:42366) 2024-05-15T12:43:10.927 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.919+0000 7f3845a5a640 1 -- 172.21.15.117:0/939648132 learned_addr learned my addr 172.21.15.117:0/939648132 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:43:10.927 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.919+0000 7f3845a5a640 10 monclient(hunting): get_auth_request con 0x7f38400b4d00 auth_method 0 2024-05-15T12:43:10.927 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.919+0000 7f3845a5a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:10.927 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.919+0000 7f3845a5a640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:10.927 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.919+0000 7f3845a5a640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:10.927 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.919+0000 7f801ca06640 1 --2- 172.21.15.117:0/284341976 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f801002e200 0x7f801002e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:10.928 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.920+0000 7f6d477fe640 1 --2- 172.21.15.117:0/4092359090 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d400071d0 0x7f6d40030830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:10.928 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.920+0000 7f801ca06640 10 monclient(hunting): get_auth_request con 0x7f801002e200 auth_method 0 2024-05-15T12:43:10.928 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.920+0000 7f801ca06640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:10.928 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.920+0000 7f801ca06640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:10.928 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.920+0000 7f801ca06640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:10.928 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.920+0000 7f3845a5a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:10.929 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.920+0000 7f3845a5a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:10.929 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.920+0000 7f3845a5a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:10.929 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.920+0000 7f8017fff640 10 monclient(hunting): handle_auth_done global_id 4284 payload 274 2024-05-15T12:43:10.929 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.920+0000 7f8017fff640 1 -- 172.21.15.117:0/284341976 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f801002e200 msgr2=0x7f801002e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:43:10.929 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.920+0000 7f8017fff640 1 --2- 172.21.15.117:0/284341976 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f801002e200 0x7f801002e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:10.929 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.920+0000 7f8017fff640 1 -- 172.21.15.117:0/284341976 >> v1:172.21.15.117:6789/0 conn(0x7f8010032db0 legacy=0x7f80100cc080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:10.929 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.920+0000 7f8017fff640 10 monclient: _finish_hunting 0 2024-05-15T12:43:10.929 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.920+0000 7f8017fff640 1 monclient: found mon.noname-a 2024-05-15T12:43:10.929 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.920+0000 7f8017fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:10.929 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.920+0000 7f8017fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.117:3300/0 2024-05-15T12:43:10.929 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.920+0000 7f8017fff640 1 -- 172.21.15.117:0/284341976 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f801002c7a0 con 0x7f8010032970 2024-05-15T12:43:10.930 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.920+0000 7f6d477fe640 10 monclient(hunting): get_auth_request con 0x7f6d400071d0 auth_method 0 2024-05-15T12:43:10.930 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.920+0000 7f6d477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:10.930 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.920+0000 7f6d477fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:10.930 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.920+0000 7f6d477fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:10.930 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.922+0000 7f3845a5a640 10 monclient(hunting): handle_auth_done global_id 4324 payload 274 2024-05-15T12:43:10.930 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.922+0000 7f6d477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:10.931 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.922+0000 7f6d477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:10.931 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.922+0000 7f6d477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:10.931 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.922+0000 7f3845a5a640 1 -- 172.21.15.117:0/939648132 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f38400b50e0 msgr2=0x7f3840145c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:43:10.932 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.922+0000 7f3845a5a640 1 --2- 172.21.15.117:0/939648132 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f38400b50e0 0x7f3840145c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:10.932 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.922+0000 7f3845a5a640 1 -- 172.21.15.117:0/939648132 >> v1:172.21.15.178:6789/0 conn(0x7f38400a94b0 legacy=0x7f38400b0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:43:10.932 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.922+0000 7f3845a5a640 10 monclient: _finish_hunting 0 2024-05-15T12:43:10.932 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.922+0000 7f3845a5a640 1 monclient: found mon.noname-c 2024-05-15T12:43:10.932 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.922+0000 7f3845a5a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:10.932 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.922+0000 7f3845a5a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.178:3300/0 2024-05-15T12:43:10.932 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.922+0000 7f3845a5a640 1 -- 172.21.15.117:0/939648132 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3840146160 con 0x7f38400b4d00 2024-05-15T12:43:10.932 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.923+0000 7f091ffff640 1 --2- 172.21.15.117:0/2594824507 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f09200b4d00 0x7f09200b1e80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f09200b3f10 tx=0x7f091002b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c44aa3d6ef75efc server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:10.933 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.923+0000 7f657ffff640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f65780cfae0 0x7f65780c5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:10.934 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.925+0000 7f6d477fe640 10 monclient(hunting): handle_auth_done global_id 4271 payload 274 2024-05-15T12:43:10.934 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.925+0000 7f6d477fe640 1 -- 172.21.15.117:0/4092359090 >> v1:172.21.15.178:6789/0 conn(0x7f6d40033560 legacy=0x7f6d40033a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:10.934 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.925+0000 7f6d477fe640 1 -- 172.21.15.117:0/4092359090 >> v1:172.21.15.117:6789/0 conn(0x7f6d400cfab0 legacy=0x7f6d400cfef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:43:10.934 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.925+0000 7f6d477fe640 10 monclient: _finish_hunting 0 2024-05-15T12:43:10.934 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.925+0000 7f6d477fe640 1 monclient: found mon.noname-b 2024-05-15T12:43:10.934 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.925+0000 7f6d477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:10.934 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.925+0000 7f6d477fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.117:3301/0 2024-05-15T12:43:10.934 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.925+0000 7f6d477fe640 1 -- 172.21.15.117:0/4092359090 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d40038db0 con 0x7f6d400071d0 2024-05-15T12:43:10.934 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.926+0000 7f07df2e3640 10 monclient: get_monmap_and_config 2024-05-15T12:43:10.934 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.926+0000 7f07df2e3640 10 monclient: build_initial_monmap 2024-05-15T12:43:10.935 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.926+0000 7f07df2e3640 10 monclient: monmap: 2024-05-15T12:43:10.935 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:10.935 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:10.935 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:43:10.927449+0000 2024-05-15T12:43:10.935 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:43:10.927449+0000 2024-05-15T12:43:10.935 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:10.935 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:10.935 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T12:43:10.935 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T12:43:10.935 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T12:43:10.935 INFO:teuthology.orchestra.run.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T12:43:10.935 INFO:teuthology.orchestra.run.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T12:43:10.935 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:10.936 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.926+0000 7f9e24fd5640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9e20066ec0 0x7f9e20145b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:10.936 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.926+0000 7f9e24fd5640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9e20066ec0 0x7f9e20145b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.117:48886/0 (socket says 172.21.15.117:48886) 2024-05-15T12:43:10.936 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.926+0000 7f9e24fd5640 1 -- 172.21.15.117:0/3422287661 learned_addr learned my addr 172.21.15.117:0/3422287661 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:43:10.936 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.926+0000 7f9e24fd5640 10 monclient(hunting): get_auth_request con 0x7f9e20066ec0 auth_method 0 2024-05-15T12:43:10.936 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.926+0000 7f9e24fd5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:10.936 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.926+0000 7f9e24fd5640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:10.936 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.926+0000 7f9e24fd5640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:10.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.927+0000 7f07df2e3640 1 Processor -- start 2024-05-15T12:43:10.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.927+0000 7f07df2e3640 1 -- start start 2024-05-15T12:43:10.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.927+0000 7f07df2e3640 10 monclient: init 2024-05-15T12:43:10.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.927+0000 7f9943fff640 1 --2- 172.21.15.117:0/363281595 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f99440716a0 0x7f994414d790 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f993800b410 tx=0x7f993800b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:10.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.927+0000 7f99417fa640 10 client.?.objecter ms_handle_connect 0x7f99440716a0 2024-05-15T12:43:10.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.927+0000 7f99417fa640 10 client.?.objecter resend_mon_ops 2024-05-15T12:43:10.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.927+0000 7f07df2e3640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:10.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.928+0000 7f07df2e3640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-05-15T12:43:10.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.928+0000 7f07df2e3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f07d80b4d00 addr v1:172.21.15.178:6789/0 2024-05-15T12:43:10.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.928+0000 7f657ffff640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f65780cfae0 0x7f65780c5b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:42382/0 (socket says 172.21.15.117:42382) 2024-05-15T12:43:10.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.928+0000 7f657ffff640 1 -- 172.21.15.117:0/3888178903 learned_addr learned my addr 172.21.15.117:0/3888178903 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:43:10.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.928+0000 7f657ffff640 10 monclient(hunting): get_auth_request con 0x7f65780cfae0 auth_method 0 2024-05-15T12:43:10.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.928+0000 7f657ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:10.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.928+0000 7f657ffff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:10.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.928+0000 7f657ffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:10.939 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.928+0000 7f07df2e3640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f07d80b50e0 0x7f07d80b1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:10.939 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.928+0000 7f07df2e3640 10 monclient(hunting): picked mon.noname-c con 0x7f07d80b50e0 addr v2:172.21.15.178:3300/0 2024-05-15T12:43:10.939 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.928+0000 7f07df2e3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f07d80a94b0 addr v1:172.21.15.117:6789/0 2024-05-15T12:43:10.939 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.928+0000 7f07df2e3640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07d80ab1b0 con 0x7f07d80a94b0 2024-05-15T12:43:10.939 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.928+0000 7f07df2e3640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07d80a9960 con 0x7f07d80b4d00 2024-05-15T12:43:10.939 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.928+0000 7f07df2e3640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:10.939 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.928+0000 7f07df2e3640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07d80b23c0 con 0x7f07d80b50e0 2024-05-15T12:43:10.940 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.928+0000 7f07df2e3640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:10.940 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.928+0000 7f07df2e3640 10 monclient(hunting): authenticate will time out at 1354.169936s 2024-05-15T12:43:10.940 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.928+0000 7f9e24fd5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:10.940 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.928+0000 7f9e24fd5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:10.940 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.928+0000 7f9e24fd5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:10.940 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.929+0000 7f657ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:10.940 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.929+0000 7f657ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:10.940 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.929+0000 7f657ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:10.941 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.932+0000 7f9e24fd5640 10 monclient(hunting): handle_auth_done global_id 4274 payload 274 2024-05-15T12:43:10.941 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.932+0000 7f9e24fd5640 1 -- 172.21.15.117:0/3422287661 >> v2:172.21.15.117:3300/0 conn(0x7f9e200adce0 msgr2=0x7f9e20146060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:43:10.942 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.932+0000 7f9e24fd5640 1 --2- 172.21.15.117:0/3422287661 >> v2:172.21.15.117:3300/0 conn(0x7f9e200adce0 0x7f9e20146060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:10.942 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.932+0000 7f9e24fd5640 1 -- 172.21.15.117:0/3422287661 >> v1:172.21.15.117:6789/0 conn(0x7f9e20066ae0 legacy=0x7f9e2013c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:43:10.942 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.932+0000 7f9e24fd5640 10 monclient: _finish_hunting 0 2024-05-15T12:43:10.942 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.932+0000 7f9e24fd5640 1 monclient: found mon.noname-b 2024-05-15T12:43:10.942 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.932+0000 7f9e24fd5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:10.942 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.933+0000 7f9e24fd5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.117:3301/0 2024-05-15T12:43:10.942 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.933+0000 7f9e24fd5640 1 -- 172.21.15.117:0/3422287661 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e200ae290 con 0x7f9e20066ec0 2024-05-15T12:43:10.942 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.933+0000 7f3845a5a640 1 --2- 172.21.15.117:0/939648132 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38400b4d00 0x7f38400b1e80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f383c0098a0 tx=0x7f383c02b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65fa7e3128fd3e03 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:10.943 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.934+0000 7f6d477fe640 1 --2- 172.21.15.117:0/4092359090 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d400071d0 0x7f6d40030830 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c0097f0 tx=0x7f6d3c02b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c549b81ac0882204 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:10.943 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.937+0000 7f657effd640 1 --2- 172.21.15.117:0/3888178903 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6578033590 0x7f657802eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:10.943 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.938+0000 7f657effd640 10 monclient(hunting): get_auth_request con 0x7f6578033590 auth_method 0 2024-05-15T12:43:10.943 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.938+0000 7f657effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:10.943 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.938+0000 7f657effd640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:10.943 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.938+0000 7f657effd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:10.944 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.941+0000 7f9e24fd5640 1 --2- 172.21.15.117:0/3422287661 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9e20066ec0 0x7f9e20145b20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9e10009870 tx=0x7f9e1002b7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67bcc67b16240335 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:10.944 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.941+0000 7f091effd640 1 -- 172.21.15.117:0/2594824507 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f091002e070 con 0x7f09200b4d00 2024-05-15T12:43:10.944 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.941+0000 7f091effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:10.944 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.941+0000 7f091effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:43:10.944 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.941+0000 7f091effd640 10 monclient: dump: 2024-05-15T12:43:10.944 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:10.944 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:10.944 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:10.944 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:10.944 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:10.945 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:10.945 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:10.945 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:10.945 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:10.945 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:10.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.941+0000 7f091effd640 10 monclient: _finish_auth 0 2024-05-15T12:43:10.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.941+0000 7f091effd640 10 monclient: _check_auth_tickets 2024-05-15T12:43:10.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.941+0000 7f091effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:10.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.941+0000 7f091effd640 1 -- 172.21.15.117:0/2594824507 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0910037070 con 0x7f09200b4d00 2024-05-15T12:43:10.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.941+0000 7f091effd640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:10.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.941+0000 7f091effd640 1 -- 172.21.15.117:0/2594824507 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0910031a00 con 0x7f09200b4d00 2024-05-15T12:43:10.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.941+0000 7f09266ef640 5 monclient: authenticate success, global_id 4321 2024-05-15T12:43:10.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.941+0000 7f09266ef640 10 monclient: get_monmap_and_config success 2024-05-15T12:43:10.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.941+0000 7f09266ef640 4 set_mon_vals no callback set 2024-05-15T12:43:10.946 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.941+0000 7f091effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:10.946 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.941+0000 7f091effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:43:10.946 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.941+0000 7f091effd640 10 monclient: dump: 2024-05-15T12:43:10.946 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:10.946 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:10.946 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:10.946 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:10.946 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:10.946 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:10.946 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:10.946 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:10.946 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:10.946 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:10.947 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.941+0000 7f657effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:10.947 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.941+0000 7f657effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:10.947 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.941+0000 7f657effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:10.947 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.942+0000 7f3844a58640 1 -- 172.21.15.117:0/939648132 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f383c0317c0 con 0x7f38400b4d00 2024-05-15T12:43:10.947 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.942+0000 7f3844a58640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:10.948 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.942+0000 7f3844a58640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:43:10.948 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.942+0000 7f3844a58640 10 monclient: dump: 2024-05-15T12:43:10.948 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:10.948 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:10.948 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:10.948 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:10.948 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:10.948 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:10.948 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:10.948 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:10.948 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:10.948 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:10.948 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.942+0000 7f3844a58640 10 monclient: _finish_auth 0 2024-05-15T12:43:10.948 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.942+0000 7f3844a58640 10 monclient: _check_auth_tickets 2024-05-15T12:43:10.948 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.942+0000 7f3844a58640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:10.948 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.942+0000 7f3847cbe640 5 monclient: authenticate success, global_id 4324 2024-05-15T12:43:10.949 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.942+0000 7f3847cbe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:43:10.949 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.942+0000 7f3844a58640 1 -- 172.21.15.117:0/939648132 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f383c037070 con 0x7f38400b4d00 2024-05-15T12:43:10.949 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.942+0000 7f3844a58640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:10.949 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.942+0000 7f3847cbe640 10 monclient: get_monmap_and_config success 2024-05-15T12:43:10.949 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.942+0000 7f3847cbe640 4 set_mon_vals no callback set 2024-05-15T12:43:10.949 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.942+0000 7f09266ef640 10 monclient: shutdown 2024-05-15T12:43:10.949 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.942+0000 7f09266ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:10.949 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.942+0000 7f09266ef640 1 -- 172.21.15.117:0/2594824507 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f09200b4d00 msgr2=0x7f09200b1e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:10.949 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.942+0000 7f09266ef640 1 --2- 172.21.15.117:0/2594824507 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f09200b4d00 0x7f09200b1e80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f09200b3f10 tx=0x7f091002b880 comp rx=0 tx=0).stop 2024-05-15T12:43:10.950 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.942+0000 7f09266ef640 1 -- 172.21.15.117:0/2594824507 shutdown_connections 2024-05-15T12:43:10.950 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.942+0000 7f09266ef640 1 --2- 172.21.15.117:0/2594824507 >> v2:172.21.15.117:3300/0 conn(0x7f09200b50e0 0x7f0920145c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:10.950 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.942+0000 7f09266ef640 1 --2- 172.21.15.117:0/2594824507 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f09200b4d00 0x7f09200b1e80 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:10.950 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.942+0000 7f09266ef640 1 -- 172.21.15.117:0/2594824507 >> 172.21.15.117:0/2594824507 conn(0x7f0920066cc0 msgr2=0x7f09200670d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:10.950 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.942+0000 7f3844a58640 1 -- 172.21.15.117:0/939648132 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f383c0317c0 con 0x7f38400b4d00 2024-05-15T12:43:10.950 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.942+0000 7f3844a58640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:10.950 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.942+0000 7f3844a58640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:43:10.950 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.942+0000 7f3844a58640 10 monclient: dump: 2024-05-15T12:43:10.950 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:10.951 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:10.951 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:10.951 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:10.951 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:10.951 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:10.951 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:10.951 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:10.951 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:10.951 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:10.951 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.942+0000 7f09266ef640 1 -- 172.21.15.117:0/2594824507 shutdown_connections 2024-05-15T12:43:10.952 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.942+0000 7f09266ef640 1 -- 172.21.15.117:0/2594824507 wait complete. 2024-05-15T12:43:10.952 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.942+0000 7f99417fa640 1 -- 172.21.15.117:0/363281595 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9938015020 con 0x7f99440716a0 2024-05-15T12:43:10.953 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.942+0000 7f99417fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:10.953 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.942+0000 7f99417fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:43:10.953 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.942+0000 7f99417fa640 10 monclient: dump: 2024-05-15T12:43:10.953 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:10.953 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:10.953 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:10.953 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:10.953 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:10.953 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:10.953 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:10.953 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:10.953 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:10.953 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:10.954 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.942+0000 7f99417fa640 10 monclient: _finish_auth 0 2024-05-15T12:43:10.954 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.942+0000 7f99417fa640 10 monclient: _check_auth_tickets 2024-05-15T12:43:10.954 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.942+0000 7f99417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:10.954 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.942+0000 7f99417fa640 1 -- 172.21.15.117:0/363281595 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9938011850 con 0x7f99440716a0 2024-05-15T12:43:10.954 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.942+0000 7f99417fa640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:10.954 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.942+0000 7f99417fa640 1 -- 172.21.15.117:0/363281595 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9938011c40 con 0x7f99440716a0 2024-05-15T12:43:10.954 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.942+0000 7f99417fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:10.954 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.942+0000 7f99417fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:43:10.954 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.942+0000 7f99417fa640 10 monclient: dump: 2024-05-15T12:43:10.954 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:10.954 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:10.954 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:10.954 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:10.954 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:10.954 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:10.955 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:10.955 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:10.955 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:10.955 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:10.955 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.942+0000 7f994aed6640 5 monclient: authenticate success, global_id 4318 2024-05-15T12:43:10.955 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.942+0000 7f994aed6640 10 monclient: _renew_subs 2024-05-15T12:43:10.955 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.942+0000 7f994aed6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:43:10.955 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.942+0000 7f994aed6640 1 -- 172.21.15.117:0/363281595 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f994414d010 con 0x7f99440716a0 2024-05-15T12:43:10.955 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.942+0000 7f9942ffd640 4 set_mon_vals no callback set 2024-05-15T12:43:10.955 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.942+0000 7f3847cbe640 10 monclient: shutdown 2024-05-15T12:43:10.955 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.942+0000 7f3847cbe640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:10.956 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.942+0000 7f994aed6640 10 client.4318.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:43:10.956 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.942+0000 7f3847cbe640 1 -- 172.21.15.117:0/939648132 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38400b4d00 msgr2=0x7f38400b1e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:10.956 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.942+0000 7f3847cbe640 1 --2- 172.21.15.117:0/939648132 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38400b4d00 0x7f38400b1e80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f383c0098a0 tx=0x7f383c02b860 comp rx=0 tx=0).stop 2024-05-15T12:43:10.957 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.942+0000 7f994aed6640 10 monclient: _renew_subs 2024-05-15T12:43:10.957 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.943+0000 7f994aed6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:43:10.957 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.943+0000 7f994aed6640 1 -- 172.21.15.117:0/363281595 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99440acf40 con 0x7f99440716a0 2024-05-15T12:43:10.957 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.943+0000 7f994aed6640 1 librados: init done 2024-05-15T12:43:10.957 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.943+0000 7f3847cbe640 1 -- 172.21.15.117:0/939648132 shutdown_connections 2024-05-15T12:43:10.957 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.943+0000 7f3847cbe640 1 --2- 172.21.15.117:0/939648132 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f38400b50e0 0x7f3840145c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:10.957 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.943+0000 7f3847cbe640 1 --2- 172.21.15.117:0/939648132 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38400b4d00 0x7f38400b1e80 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:10.957 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.943+0000 7f3847cbe640 1 -- 172.21.15.117:0/939648132 >> 172.21.15.117:0/939648132 conn(0x7f3840066cc0 msgr2=0x7f38400670d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:10.958 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.943+0000 7f3847cbe640 1 -- 172.21.15.117:0/939648132 shutdown_connections 2024-05-15T12:43:10.958 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.943+0000 7f3847cbe640 1 -- 172.21.15.117:0/939648132 wait complete. 2024-05-15T12:43:10.958 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.943+0000 7f09266ef640 10 monclient: build_initial_monmap 2024-05-15T12:43:10.958 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.943+0000 7f09266ef640 10 monclient: monmap: 2024-05-15T12:43:10.958 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:10.958 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:43:10.959 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-05-15T12:43:10.959 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-05-15T12:43:10.959 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:10.959 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:10.959 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T12:43:10.959 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T12:43:10.959 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T12:43:10.959 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:10.959 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.943+0000 7f09266ef640 1 librados: starting msgr at 2024-05-15T12:43:10.959 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.943+0000 7f09266ef640 1 librados: starting objecter 2024-05-15T12:43:10.959 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.943+0000 7f09266ef640 1 Processor -- start 2024-05-15T12:43:10.959 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.943+0000 7f3847cbe640 10 monclient: build_initial_monmap 2024-05-15T12:43:10.960 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.943+0000 7f3847cbe640 10 monclient: monmap: 2024-05-15T12:43:10.960 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:10.960 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:43:10.960 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-05-15T12:43:10.960 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-05-15T12:43:10.960 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:10.960 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:10.960 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T12:43:10.960 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T12:43:10.960 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T12:43:10.960 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:10.960 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.944+0000 7f994aed6640 10 client.4318.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:10.961 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.944+0000 7f09266ef640 1 -- start start 2024-05-15T12:43:10.961 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.944+0000 7f09266ef640 1 librados: setting wanted keys 2024-05-15T12:43:10.961 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.944+0000 7f09266ef640 1 librados: calling monclient init 2024-05-15T12:43:10.961 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.944+0000 7f09266ef640 10 monclient: init 2024-05-15T12:43:10.961 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.944+0000 7f994aed6640 15 client.4318.objecter _session_command_op_assign -1 1 2024-05-15T12:43:10.961 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.944+0000 7f994aed6640 20 client.4318.objecter _get_session osd=-1 returning homeless 2024-05-15T12:43:10.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.944+0000 7f994aed6640 10 client.4318.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:43:10.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.944+0000 7f994aed6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:43:10.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.944+0000 7f994aed6640 1 -- 172.21.15.117:0/363281595 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f99440ad200 con 0x7f99440716a0 2024-05-15T12:43:10.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.944+0000 7f3847cbe640 1 librados: starting msgr at 2024-05-15T12:43:10.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.944+0000 7f3847cbe640 1 librados: starting objecter 2024-05-15T12:43:10.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.944+0000 7f3847cbe640 1 Processor -- start 2024-05-15T12:43:10.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.944+0000 7f3847cbe640 1 -- start start 2024-05-15T12:43:10.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.944+0000 7f3847cbe640 1 librados: setting wanted keys 2024-05-15T12:43:10.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.944+0000 7f3847cbe640 1 librados: calling monclient init 2024-05-15T12:43:10.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.944+0000 7f3847cbe640 10 monclient: init 2024-05-15T12:43:10.963 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.944+0000 7f09266ef640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:10.963 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.944+0000 7f09266ef640 10 monclient: _add_conns ranks=[0,2,1] 2024-05-15T12:43:10.963 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.944+0000 7f09266ef640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f09200a94b0 0x7f092013b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:10.963 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.944+0000 7f09266ef640 10 monclient(hunting): picked mon.noname-a con 0x7f09200a94b0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T12:43:10.963 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.944+0000 7f09266ef640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f09200b4d00 0x7f09201db4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:10.963 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.944+0000 7f09266ef640 10 monclient(hunting): picked mon.noname-c con 0x7f09200b4d00 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:43:10.964 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.944+0000 7f09266ef640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f09201dc0a0 0x7f092013e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:10.964 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.944+0000 7f09266ef640 10 monclient(hunting): picked mon.noname-b con 0x7f09201dc0a0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:43:10.964 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.944+0000 7f09266ef640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:10.964 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.944+0000 7f09266ef640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f092014cfb0 con 0x7f09200a94b0 2024-05-15T12:43:10.964 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.944+0000 7f09266ef640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:10.964 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.944+0000 7f09266ef640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09200a9b70 con 0x7f09201dc0a0 2024-05-15T12:43:10.964 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.944+0000 7f09266ef640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:10.964 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.944+0000 7f09266ef640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f092014d130 con 0x7f09200b4d00 2024-05-15T12:43:10.964 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.944+0000 7f09266ef640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:10.964 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.944+0000 7f09266ef640 10 monclient(hunting): authenticate will time out at 1354.186556s 2024-05-15T12:43:10.965 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.945+0000 7f657effd640 10 monclient(hunting): handle_auth_done global_id 4277 payload 824 2024-05-15T12:43:10.965 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.945+0000 7f657effd640 1 -- 172.21.15.117:0/3888178903 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f65780cfae0 msgr2=0x7f65780c5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:10.965 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.945+0000 7f657effd640 1 --2- 172.21.15.117:0/3888178903 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f65780cfae0 0x7f65780c5b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:10.965 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.945+0000 7f657effd640 1 -- 172.21.15.117:0/3888178903 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6578007200 msgr2=0x7f6578164b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:10.965 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.945+0000 7f657effd640 1 --2- 172.21.15.117:0/3888178903 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6578007200 0x7f6578164b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:10.965 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.945+0000 7f657effd640 10 monclient: _finish_hunting 0 2024-05-15T12:43:10.965 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.945+0000 7f657effd640 1 monclient: found mon.noname-c 2024-05-15T12:43:10.965 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.945+0000 7f657effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:10.965 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.945+0000 7f657effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.117:3301/0 2024-05-15T12:43:10.965 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.945+0000 7f657effd640 1 -- 172.21.15.117:0/3888178903 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6578009d80 con 0x7f6578033590 2024-05-15T12:43:10.966 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.945+0000 7f3847cbe640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:10.966 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.945+0000 7f3847cbe640 10 monclient: _add_conns ranks=[0,1,2] 2024-05-15T12:43:10.966 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.945+0000 7f3847cbe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f38400a94b0 0x7f3840142800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:10.966 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.945+0000 7f3847cbe640 10 monclient(hunting): picked mon.noname-a con 0x7f38400a94b0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T12:43:10.966 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.945+0000 7f3847cbe640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38400b4d00 0x7f38401df600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:10.966 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.945+0000 7f3847cbe640 10 monclient(hunting): picked mon.noname-b con 0x7f38400b4d00 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:43:10.966 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.945+0000 7f3847cbe640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f384013dac0 0x7f384013ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:10.966 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.945+0000 7f3847cbe640 10 monclient(hunting): picked mon.noname-c con 0x7f384013dac0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:43:10.966 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.945+0000 7f3847cbe640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:10.966 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.945+0000 7f3847cbe640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f384014cfb0 con 0x7f38400a94b0 2024-05-15T12:43:10.967 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.945+0000 7f3847cbe640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:10.967 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.945+0000 7f3847cbe640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f384014d130 con 0x7f38400b4d00 2024-05-15T12:43:10.967 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.945+0000 7f3847cbe640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:10.967 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.945+0000 7f3847cbe640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f384014d2b0 con 0x7f384013dac0 2024-05-15T12:43:10.967 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.945+0000 7f3847cbe640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:10.967 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.945+0000 7f3847cbe640 10 monclient(hunting): authenticate will time out at 1354.187286s 2024-05-15T12:43:10.967 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.950+0000 7f8017fff640 1 --2- 172.21.15.117:0/284341976 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8010032970 0x7f80100cf580 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f80040097a0 tx=0x7f800402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7ab7ab4b75f9572 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:10.967 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.950+0000 7f384625b640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f384013dac0 0x7f384013ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:10.968 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.950+0000 7f091f7fe640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f09200b4d00 0x7f09201db4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:10.968 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.950+0000 7f384625b640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f384013dac0 0x7f384013ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.117:48908/0 (socket says 172.21.15.117:48908) 2024-05-15T12:43:10.968 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.950+0000 7f384625b640 1 -- 172.21.15.117:0/1662615750 learned_addr learned my addr 172.21.15.117:0/1662615750 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:43:10.968 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.950+0000 7f384625b640 10 monclient(hunting): get_auth_request con 0x7f384013dac0 auth_method 0 2024-05-15T12:43:10.968 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.950+0000 7f384625b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:10.968 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.950+0000 7f384625b640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:10.968 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.950+0000 7f384625b640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:10.969 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.951+0000 7f657effd640 1 --2- 172.21.15.117:0/3888178903 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6578033590 0x7f657802eab0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f657400b410 tx=0x7f657400b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:10.969 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.951+0000 7f656bfff640 10 client.?.objecter ms_handle_connect 0x7f6578033590 2024-05-15T12:43:10.969 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.951+0000 7f656bfff640 10 client.?.objecter resend_mon_ops 2024-05-15T12:43:10.969 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.952+0000 7f384625b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:10.969 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.952+0000 7f384625b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:10.969 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.952+0000 7f384625b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:10.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.953+0000 7f07dd880640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f07d80a94b0 0x7f07d80b0fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:48668/0 (socket says 172.21.15.117:48668) 2024-05-15T12:43:10.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.953+0000 7f07dd880640 1 -- 172.21.15.117:0/1548430383 learned_addr learned my addr 172.21.15.117:0/1548430383 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-05-15T12:43:10.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.955+0000 7f384625b640 10 monclient(hunting): handle_auth_done global_id 4280 payload 824 2024-05-15T12:43:10.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.955+0000 7f384625b640 1 -- 172.21.15.117:0/1662615750 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38400b4d00 msgr2=0x7f38401df600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:10.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.955+0000 7f384625b640 1 --2- 172.21.15.117:0/1662615750 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38400b4d00 0x7f38401df600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:10.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.955+0000 7f384625b640 1 -- 172.21.15.117:0/1662615750 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f38400a94b0 msgr2=0x7f3840142800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:10.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.955+0000 7f384625b640 1 --2- 172.21.15.117:0/1662615750 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f38400a94b0 0x7f3840142800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:10.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.955+0000 7f384625b640 10 monclient: _finish_hunting 0 2024-05-15T12:43:10.971 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.955+0000 7f384625b640 1 monclient: found mon.noname-c 2024-05-15T12:43:10.971 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.955+0000 7f384625b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:10.971 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.955+0000 7f384625b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.117:3301/0 2024-05-15T12:43:10.971 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.955+0000 7f384625b640 1 -- 172.21.15.117:0/1662615750 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38401dfb40 con 0x7f384013dac0 2024-05-15T12:43:10.971 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.955+0000 7f091f7fe640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f09200b4d00 0x7f09201db4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.117:48894/0 (socket says 172.21.15.117:48894) 2024-05-15T12:43:10.972 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.955+0000 7f091f7fe640 1 -- 172.21.15.117:0/2876865113 learned_addr learned my addr 172.21.15.117:0/2876865113 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:43:10.972 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.955+0000 7f091f7fe640 10 monclient(hunting): get_auth_request con 0x7f09200b4d00 auth_method 0 2024-05-15T12:43:10.972 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.955+0000 7f091f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:10.972 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.955+0000 7f091f7fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:10.972 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.955+0000 7f091f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:10.972 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.957+0000 7f091f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:10.972 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.957+0000 7f091f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:10.972 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.957+0000 7f091f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:10.972 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.958+0000 7f091ffff640 1 --2- 172.21.15.117:0/2876865113 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f09200a94b0 0x7f092013b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:10.972 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.959+0000 7f091ffff640 10 monclient(hunting): get_auth_request con 0x7f09200a94b0 auth_method 0 2024-05-15T12:43:10.972 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.959+0000 7f091ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:10.972 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.959+0000 7f091ffff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:10.972 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.959+0000 7f091ffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:10.973 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.959+0000 7f07dc87e640 1 --2- 172.21.15.117:0/1548430383 >> v2:172.21.15.178:3300/0 conn(0x7f07d80b50e0 0x7f07d80b1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:10.973 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.960+0000 7f07dc87e640 10 monclient(hunting): get_auth_request con 0x7f07d80b50e0 auth_method 0 2024-05-15T12:43:10.973 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.960+0000 7f07dc87e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:10.973 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.960+0000 7f07dc87e640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:10.973 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.960+0000 7f07dc87e640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:10.974 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.961+0000 7f80177fe640 1 -- 172.21.15.117:0/284341976 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8004034070 con 0x7f8010032970 2024-05-15T12:43:10.974 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.961+0000 7f80177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:10.974 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.961+0000 7f80177fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T12:43:10.974 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.961+0000 7f80177fe640 10 monclient: dump: 2024-05-15T12:43:10.974 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:10.974 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:10.974 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:10.974 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:10.975 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:10.975 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:10.975 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:10.975 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:10.975 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:10.975 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:10.975 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.961+0000 7f80177fe640 10 monclient: _finish_auth 0 2024-05-15T12:43:10.975 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.961+0000 7f80177fe640 10 monclient: _check_auth_tickets 2024-05-15T12:43:10.975 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.961+0000 7f80177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:10.975 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.961+0000 7f801da08640 5 monclient: authenticate success, global_id 4284 2024-05-15T12:43:10.975 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.961+0000 7f801da08640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:43:10.975 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.961+0000 7f80177fe640 1 -- 172.21.15.117:0/284341976 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8004030610 con 0x7f8010032970 2024-05-15T12:43:10.975 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.961+0000 7f80177fe640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:10.975 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.961+0000 7f801da08640 10 monclient: get_monmap_and_config success 2024-05-15T12:43:10.976 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.961+0000 7f801da08640 4 set_mon_vals no callback set 2024-05-15T12:43:10.976 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.961+0000 7f0924c8c640 1 --2- 172.21.15.117:0/2876865113 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f09201dc0a0 0x7f092013e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:10.976 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.961+0000 7f80177fe640 1 -- 172.21.15.117:0/284341976 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80040309e0 con 0x7f8010032970 2024-05-15T12:43:10.976 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.961+0000 7f80177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:10.976 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.961+0000 7f80177fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T12:43:10.976 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.961+0000 7f80177fe640 10 monclient: dump: 2024-05-15T12:43:10.977 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:10.977 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:10.977 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:10.977 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:10.977 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:10.977 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:10.977 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:10.977 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:10.977 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:10.977 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:10.978 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.961+0000 7f07dc87e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:10.978 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.961+0000 7f07dc87e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:10.978 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.961+0000 7f07dc87e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:10.978 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.962+0000 7f801da08640 10 monclient: shutdown 2024-05-15T12:43:10.978 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.962+0000 7f801da08640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:10.978 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.962+0000 7f801da08640 1 -- 172.21.15.117:0/284341976 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8010032970 msgr2=0x7f80100cf580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:10.978 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.962+0000 7f801da08640 1 --2- 172.21.15.117:0/284341976 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8010032970 0x7f80100cf580 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f80040097a0 tx=0x7f800402b960 comp rx=0 tx=0).stop 2024-05-15T12:43:10.979 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.962+0000 7f0924c8c640 10 monclient(hunting): get_auth_request con 0x7f09201dc0a0 auth_method 0 2024-05-15T12:43:10.979 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.962+0000 7f0924c8c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:10.979 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.962+0000 7f0924c8c640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:10.979 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.962+0000 7f0924c8c640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:10.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.962+0000 7f801da08640 1 -- 172.21.15.117:0/284341976 shutdown_connections 2024-05-15T12:43:10.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.962+0000 7f801da08640 1 --2- 172.21.15.117:0/284341976 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8010032970 0x7f80100cf580 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:10.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.962+0000 7f801da08640 1 --2- 172.21.15.117:0/284341976 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f801002e200 0x7f801002e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:10.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.962+0000 7f801da08640 1 -- 172.21.15.117:0/284341976 >> 172.21.15.117:0/284341976 conn(0x7f8010036620 msgr2=0x7f8010005e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:10.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.962+0000 7f801da08640 1 -- 172.21.15.117:0/284341976 shutdown_connections 2024-05-15T12:43:10.981 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.962+0000 7f091ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:10.981 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.962+0000 7f091ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:10.981 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.962+0000 7f091ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:10.982 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.962+0000 7f801da08640 1 -- 172.21.15.117:0/284341976 wait complete. 2024-05-15T12:43:10.982 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.962+0000 7f07cffff640 1 -- 172.21.15.117:0/1548430383 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3060282844 0 0) 0x7f07d80ab1b0 con 0x7f07d80a94b0 2024-05-15T12:43:10.982 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.962+0000 7f07cffff640 10 monclient(hunting): _negotiate 2024-05-15T12:43:10.982 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.962+0000 7f07cffff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:10.982 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.962+0000 7f07cffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:10.982 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.962+0000 7f07cffff640 10 monclient(hunting): my global_id is 4287 2024-05-15T12:43:10.982 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.962+0000 7f07cffff640 1 -- 172.21.15.117:0/1548430383 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07c4003520 con 0x7f07d80a94b0 2024-05-15T12:43:10.983 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.962+0000 7f091f7fe640 10 monclient(hunting): handle_auth_done global_id 4283 payload 824 2024-05-15T12:43:10.983 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.962+0000 7f091f7fe640 1 -- 172.21.15.117:0/2876865113 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f09201dc0a0 msgr2=0x7f092013e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:10.983 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.962+0000 7f091f7fe640 1 --2- 172.21.15.117:0/2876865113 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f09201dc0a0 0x7f092013e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:10.983 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.962+0000 7f091f7fe640 1 -- 172.21.15.117:0/2876865113 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f09200a94b0 msgr2=0x7f092013b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:10.983 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.962+0000 7f091f7fe640 1 --2- 172.21.15.117:0/2876865113 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f09200a94b0 0x7f092013b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:10.983 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.962+0000 7f091f7fe640 10 monclient: _finish_hunting 0 2024-05-15T12:43:10.983 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.962+0000 7f091f7fe640 1 monclient: found mon.noname-c 2024-05-15T12:43:10.983 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.962+0000 7f091f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:10.984 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.962+0000 7f091f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.117:3301/0 2024-05-15T12:43:10.984 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.962+0000 7f091f7fe640 1 -- 172.21.15.117:0/2876865113 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09200ac6d0 con 0x7f09200b4d00 2024-05-15T12:43:10.985 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.963+0000 7f801da08640 10 monclient: build_initial_monmap 2024-05-15T12:43:10.985 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.963+0000 7f801da08640 10 monclient: monmap: 2024-05-15T12:43:10.985 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:10.985 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:43:10.985 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-05-15T12:43:10.985 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-05-15T12:43:10.985 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:10.986 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:10.986 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T12:43:10.986 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T12:43:10.986 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T12:43:10.986 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:10.986 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.963+0000 7f801da08640 1 librados: starting msgr at 2024-05-15T12:43:10.986 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.963+0000 7f801da08640 1 librados: starting objecter 2024-05-15T12:43:10.986 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.963+0000 7f801da08640 1 Processor -- start 2024-05-15T12:43:10.986 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.963+0000 7f801da08640 1 -- start start 2024-05-15T12:43:10.986 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.963+0000 7f801da08640 1 librados: setting wanted keys 2024-05-15T12:43:10.986 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.963+0000 7f801da08640 1 librados: calling monclient init 2024-05-15T12:43:10.986 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.963+0000 7f801da08640 10 monclient: init 2024-05-15T12:43:10.986 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.964+0000 7f801da08640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:10.987 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.964+0000 7f801da08640 10 monclient: _add_conns ranks=[0,1,2] 2024-05-15T12:43:10.987 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.964+0000 7f801da08640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f801002e200 0x7f80100a3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:10.987 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.964+0000 7f801da08640 10 monclient(hunting): picked mon.noname-a con 0x7f801002e200 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T12:43:10.987 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.964+0000 7f801da08640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8010032970 0x7f80100c59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:10.987 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.964+0000 7f801da08640 10 monclient(hunting): picked mon.noname-b con 0x7f8010032970 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:43:10.987 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.964+0000 7f801da08640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8010032db0 0x7f80100ce410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:10.987 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.964+0000 7f801da08640 10 monclient(hunting): picked mon.noname-c con 0x7f8010032db0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:43:10.987 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.964+0000 7f801da08640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:10.987 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.964+0000 7f801da08640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80100cfd50 con 0x7f801002e200 2024-05-15T12:43:10.987 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.964+0000 7f801da08640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:10.987 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.964+0000 7f801da08640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f801002d8a0 con 0x7f8010032970 2024-05-15T12:43:10.987 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.964+0000 7f801da08640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:10.987 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.964+0000 7f801da08640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80100cfed0 con 0x7f8010032db0 2024-05-15T12:43:10.987 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.964+0000 7f801da08640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:10.987 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.964+0000 7f801da08640 10 monclient(hunting): authenticate will time out at 1354.206406s 2024-05-15T12:43:10.988 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.965+0000 7f384625b640 1 --2- 172.21.15.117:0/1662615750 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f384013dac0 0x7f384013ea70 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f383800ba10 tx=0x7f383800bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:10.988 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.965+0000 7f38367fc640 10 client.?.objecter ms_handle_connect 0x7f384013dac0 2024-05-15T12:43:10.988 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.965+0000 7f38367fc640 10 client.?.objecter resend_mon_ops 2024-05-15T12:43:10.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.967+0000 7f6d467fc640 1 -- 172.21.15.117:0/4092359090 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d3c035020 con 0x7f6d400071d0 2024-05-15T12:43:10.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.967+0000 7f6d467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:10.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.967+0000 7f6d467fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:43:10.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.967+0000 7f6d467fc640 10 monclient: dump: 2024-05-15T12:43:10.990 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:10.990 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:10.990 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:10.990 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:10.990 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:10.991 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:10.991 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:10.991 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:10.991 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:10.991 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:10.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.967+0000 7f6d467fc640 10 monclient: _finish_auth 0 2024-05-15T12:43:10.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.967+0000 7f6d467fc640 10 monclient: _check_auth_tickets 2024-05-15T12:43:10.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.967+0000 7f6d467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:10.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.967+0000 7f6d467fc640 1 -- 172.21.15.117:0/4092359090 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d3c031870 con 0x7f6d400071d0 2024-05-15T12:43:10.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.967+0000 7f6d4c9d3640 5 monclient: authenticate success, global_id 4271 2024-05-15T12:43:10.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.967+0000 7f6d467fc640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:10.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.967+0000 7f6d4c9d3640 10 monclient: get_monmap_and_config success 2024-05-15T12:43:10.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.967+0000 7f6d4c9d3640 4 set_mon_vals no callback set 2024-05-15T12:43:10.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.968+0000 7f6d467fc640 1 -- 172.21.15.117:0/4092359090 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d3c031c40 con 0x7f6d400071d0 2024-05-15T12:43:10.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.968+0000 7f6d467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:10.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.968+0000 7f6d467fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T12:43:10.992 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.968+0000 7f6d467fc640 10 monclient: dump: 2024-05-15T12:43:10.992 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:10.992 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:10.992 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:10.992 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:10.992 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:10.992 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:10.992 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:10.992 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:10.992 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:10.992 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:10.992 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.968+0000 7f6d4c9d3640 10 monclient: shutdown 2024-05-15T12:43:10.992 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.968+0000 7f6d4c9d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:10.992 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.968+0000 7f6d4c9d3640 1 -- 172.21.15.117:0/4092359090 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d400071d0 msgr2=0x7f6d40030830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:10.993 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.968+0000 7f6d4c9d3640 1 --2- 172.21.15.117:0/4092359090 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d400071d0 0x7f6d40030830 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c0097f0 tx=0x7f6d3c02b910 comp rx=0 tx=0).stop 2024-05-15T12:43:10.993 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.968+0000 7f6d4c9d3640 1 -- 172.21.15.117:0/4092359090 shutdown_connections 2024-05-15T12:43:10.993 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.968+0000 7f6d4c9d3640 1 --2- 172.21.15.117:0/4092359090 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d400071d0 0x7f6d40030830 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:10.993 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.968+0000 7f6d4c9d3640 1 -- 172.21.15.117:0/4092359090 >> 172.21.15.117:0/4092359090 conn(0x7f6d400061a0 msgr2=0x7f6d400065b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:10.993 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.968+0000 7f6d4c9d3640 1 -- 172.21.15.117:0/4092359090 shutdown_connections 2024-05-15T12:43:10.993 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.969+0000 7f6d4c9d3640 1 -- 172.21.15.117:0/4092359090 wait complete. 2024-05-15T12:43:10.993 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.969+0000 7f6d4c9d3640 10 monclient: build_initial_monmap 2024-05-15T12:43:10.993 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.969+0000 7f6d4c9d3640 10 monclient: monmap: 2024-05-15T12:43:10.993 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:10.993 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:43:10.993 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-05-15T12:43:10.993 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-05-15T12:43:10.993 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:10.993 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:10.993 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T12:43:10.993 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T12:43:10.993 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T12:43:10.994 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:10.994 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.970+0000 7f6d4c9d3640 1 librados: starting msgr at 2024-05-15T12:43:10.994 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.970+0000 7f6d4c9d3640 1 librados: starting objecter 2024-05-15T12:43:10.994 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.970+0000 7f6d4c9d3640 1 Processor -- start 2024-05-15T12:43:10.994 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.970+0000 7f6d4c9d3640 1 -- start start 2024-05-15T12:43:10.994 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.970+0000 7f6d4c9d3640 1 librados: setting wanted keys 2024-05-15T12:43:10.994 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.970+0000 7f6d4c9d3640 1 librados: calling monclient init 2024-05-15T12:43:10.994 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.970+0000 7f6d4c9d3640 10 monclient: init 2024-05-15T12:43:10.994 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.971+0000 7f091f7fe640 1 --2- 172.21.15.117:0/2876865113 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f09200b4d00 0x7f09201db4b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f091400b440 tx=0x7f091400b900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:10.995 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.971+0000 7f091cff9640 10 client.?.objecter ms_handle_connect 0x7f09200b4d00 2024-05-15T12:43:10.995 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.971+0000 7f091cff9640 10 client.?.objecter resend_mon_ops 2024-05-15T12:43:10.996 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.971+0000 7f6d4c9d3640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:10.996 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.971+0000 7f6d4c9d3640 10 monclient: _add_conns ranks=[1,2,0] 2024-05-15T12:43:10.996 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.971+0000 7f6d4c9d3640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6d400071d0 0x7f6d400c1d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:10.996 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.971+0000 7f6d4c9d3640 10 monclient(hunting): picked mon.noname-b con 0x7f6d400071d0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:43:10.996 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.971+0000 7f6d4c9d3640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d40033560 0x7f6d40164d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:10.996 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.971+0000 7f6d4c9d3640 10 monclient(hunting): picked mon.noname-c con 0x7f6d40033560 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:43:10.996 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.971+0000 7f6d4c9d3640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6d400cfab0 0x7f6d401652a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:10.996 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.971+0000 7f6d4c9d3640 10 monclient(hunting): picked mon.noname-a con 0x7f6d400cfab0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T12:43:10.996 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.971+0000 7f6d4c9d3640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:10.996 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.971+0000 7f6d4c9d3640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d400d7c60 con 0x7f6d400cfab0 2024-05-15T12:43:10.996 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.971+0000 7f6d4c9d3640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:10.996 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.971+0000 7f6d4c9d3640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d400d7ae0 con 0x7f6d400071d0 2024-05-15T12:43:10.996 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.971+0000 7f6d4c9d3640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:10.996 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.971+0000 7f6d4c9d3640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d400d7de0 con 0x7f6d40033560 2024-05-15T12:43:10.997 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.971+0000 7f6d4c9d3640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:10.997 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.971+0000 7f6d4c9d3640 10 monclient(hunting): authenticate will time out at 1354.213280s 2024-05-15T12:43:10.997 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.973+0000 7f656bfff640 1 -- 172.21.15.117:0/3888178903 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6574015020 con 0x7f6578033590 2024-05-15T12:43:10.997 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.973+0000 7f656bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:10.998 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.973+0000 7f656bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:43:10.998 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.973+0000 7f656bfff640 10 monclient: dump: 2024-05-15T12:43:10.998 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:10.998 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:10.998 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:10.998 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:10.998 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:10.998 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:10.998 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:10.998 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:10.998 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:10.998 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:10.998 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.973+0000 7f656bfff640 10 monclient: _finish_auth 0 2024-05-15T12:43:10.998 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.973+0000 7f656bfff640 10 monclient: _check_auth_tickets 2024-05-15T12:43:10.999 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.973+0000 7f656bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:10.999 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.973+0000 7f656bfff640 1 -- 172.21.15.117:0/3888178903 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6574011850 con 0x7f6578033590 2024-05-15T12:43:10.999 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.973+0000 7f656bfff640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:10.999 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.973+0000 7f656bfff640 1 -- 172.21.15.117:0/3888178903 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6574011c40 con 0x7f6578033590 2024-05-15T12:43:10.999 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.973+0000 7f656bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:10.999 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.973+0000 7f656bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T12:43:10.999 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.973+0000 7f656bfff640 10 monclient: dump: 2024-05-15T12:43:10.999 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:10.999 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:10.999 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:10.999 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:10.999 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:10.999 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:10.999 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:10.999 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:11.000 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:11.000 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.000 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.973+0000 7f657dffb640 4 set_mon_vals no callback set 2024-05-15T12:43:11.000 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.974+0000 7f6586164640 5 monclient: authenticate success, global_id 4277 2024-05-15T12:43:11.000 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.974+0000 7f6586164640 10 monclient: _renew_subs 2024-05-15T12:43:11.000 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.974+0000 7f6586164640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T12:43:11.000 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.974+0000 7f6586164640 1 -- 172.21.15.117:0/3888178903 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65780c6c50 con 0x7f6578033590 2024-05-15T12:43:11.000 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.974+0000 7f6586164640 10 client.4277.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:43:11.000 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.974+0000 7f6586164640 10 monclient: _renew_subs 2024-05-15T12:43:11.000 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.974+0000 7f6586164640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T12:43:11.000 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.974+0000 7f6586164640 1 -- 172.21.15.117:0/3888178903 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65781656e0 con 0x7f6578033590 2024-05-15T12:43:11.000 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.974+0000 7f6586164640 1 librados: init done 2024-05-15T12:43:11.001 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.974+0000 7f07cffff640 1 -- 172.21.15.117:0/1548430383 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3697687285 0 0) 0x7f07c4003520 con 0x7f07d80a94b0 2024-05-15T12:43:11.001 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.974+0000 7f07cffff640 1 -- 172.21.15.117:0/1548430383 >> v2:172.21.15.178:3300/0 conn(0x7f07d80b50e0 msgr2=0x7f07d80b1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:43:11.001 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.974+0000 7f07cffff640 1 --2- 172.21.15.117:0/1548430383 >> v2:172.21.15.178:3300/0 conn(0x7f07d80b50e0 0x7f07d80b1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.001 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.974+0000 7f07cffff640 1 -- 172.21.15.117:0/1548430383 >> v1:172.21.15.178:6789/0 conn(0x7f07d80b4d00 legacy=0x7f07d8145c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:43:11.001 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.974+0000 7f07cffff640 10 monclient: _finish_hunting 0 2024-05-15T12:43:11.001 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.974+0000 7f07cffff640 1 monclient: found mon.noname-a-legacy 2024-05-15T12:43:11.001 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.974+0000 7f07cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:11.001 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.974+0000 7f07cffff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.117:6789/0 2024-05-15T12:43:11.001 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.974+0000 7f07cffff640 1 -- 172.21.15.117:0/1548430383 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07d80a9b30 con 0x7f07d80a94b0 2024-05-15T12:43:11.001 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.974+0000 7f07cffff640 10 monclient: _finish_auth 0 2024-05-15T12:43:11.001 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.974+0000 7f07cffff640 10 monclient: _check_auth_tickets 2024-05-15T12:43:11.001 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.974+0000 7f07cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:11.001 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.975+0000 7f07cffff640 1 -- 172.21.15.117:0/1548430383 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3166367242 0 0) 0x7f07d00019f0 con 0x7f07d80a94b0 2024-05-15T12:43:11.002 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.975+0000 7f07cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:11.002 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.975+0000 7f07cffff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-05-15T12:43:11.002 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.975+0000 7f07cffff640 10 monclient: dump: 2024-05-15T12:43:11.002 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:11.002 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:11.002 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.002 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.002 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:11.002 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.002 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:11.002 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:11.002 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:11.002 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.002 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.975+0000 7f07cffff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] but i'm connected to v1:172.21.15.117:6789/0, reconnecting 2024-05-15T12:43:11.002 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.975+0000 7f07cffff640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:11.003 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.975+0000 7f07cffff640 1 -- 172.21.15.117:0/1548430383 >> v1:172.21.15.117:6789/0 conn(0x7f07d80a94b0 legacy=0x7f07d80b0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.003 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.975+0000 7f07cffff640 10 monclient: _add_conns ranks=[2,0,1] 2024-05-15T12:43:11.003 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.975+0000 7f07cffff640 1 --2- 172.21.15.117:0/1548430383 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f07c4003a00 0x7f07c4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.003 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.975+0000 7f07cffff640 10 monclient(hunting): picked mon.c con 0x7f07c4003a00 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:43:11.003 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.975+0000 7f07cffff640 1 --2- 172.21.15.117:0/1548430383 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f07c4004800 0x7f07c4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.003 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.975+0000 7f6586164640 10 client.4277.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:11.003 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.975+0000 7f6586164640 15 client.4277.objecter _session_command_op_assign -1 1 2024-05-15T12:43:11.003 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.975+0000 7f07cffff640 10 monclient(hunting): picked mon.a con 0x7f07c4004800 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T12:43:11.003 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.975+0000 7f07cffff640 1 --2- 172.21.15.117:0/1548430383 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f07c4005210 0x7f07c400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.004 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.975+0000 7f07cffff640 10 monclient(hunting): picked mon.b con 0x7f07c4005210 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:43:11.004 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.975+0000 7f07cffff640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:11.004 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.975+0000 7f07cffff640 1 -- 172.21.15.117:0/1548430383 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07d80b23c0 con 0x7f07c4004800 2024-05-15T12:43:11.004 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.975+0000 7f07cffff640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:11.004 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.975+0000 7f6586164640 20 client.4277.objecter _get_session osd=-1 returning homeless 2024-05-15T12:43:11.004 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.975+0000 7f6586164640 10 client.4277.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:43:11.004 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.975+0000 7f07cffff640 1 -- 172.21.15.117:0/1548430383 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07c400c140 con 0x7f07c4005210 2024-05-15T12:43:11.004 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.975+0000 7f07cffff640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:11.004 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.975+0000 7f07cffff640 1 -- 172.21.15.117:0/1548430383 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07c400c340 con 0x7f07c4003a00 2024-05-15T12:43:11.005 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.975+0000 7f07cffff640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:11.005 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.975+0000 7f07cffff640 10 monclient(hunting): _finish_auth 0 2024-05-15T12:43:11.005 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.975+0000 7f99417fa640 1 -- 172.21.15.117:0/363281595 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f9938018020 con 0x7f99440716a0 2024-05-15T12:43:11.005 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.975+0000 7f6586164640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T12:43:11.005 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.975+0000 7f6586164640 1 -- 172.21.15.117:0/3888178903 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f65780080b0 con 0x7f6578033590 2024-05-15T12:43:11.005 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.975+0000 7f99417fa640 1 --2- 172.21.15.117:0/363281595 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f993006ceb0 0x7f993006f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.006 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.975+0000 7f801ca06640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f801002e200 0x7f80100a3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:11.006 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.975+0000 7f9948c72640 1 --2- 172.21.15.117:0/363281595 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f993006ceb0 0x7f993006f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:11.006 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.975+0000 7f9948c72640 10 monclient: get_auth_request con 0x7f993006ceb0 auth_method 0 2024-05-15T12:43:11.007 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.975+0000 7f6d47fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6d400cfab0 0x7f6d401652a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:11.007 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.976+0000 7f99417fa640 10 client.4318.objecter ms_handle_connect 0x7f993006ceb0 2024-05-15T12:43:11.007 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.976+0000 7f9948c72640 1 --2- 172.21.15.117:0/363281595 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f993006ceb0 0x7f993006f370 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f993c035000 tx=0x7f993c02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:11.007 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.976+0000 7f801ca06640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f801002e200 0x7f80100a3c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:60136/0 (socket says 172.21.15.117:60136) 2024-05-15T12:43:11.007 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.976+0000 7f801ca06640 1 -- 172.21.15.117:0/3984485137 learned_addr learned my addr 172.21.15.117:0/3984485137 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:43:11.007 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.976+0000 7f801ca06640 10 monclient(hunting): get_auth_request con 0x7f801002e200 auth_method 0 2024-05-15T12:43:11.008 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.976+0000 7f801ca06640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:11.008 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.976+0000 7f801ca06640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:11.008 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.976+0000 7f801ca06640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:11.008 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.976+0000 7f6d47fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6d400cfab0 0x7f6d401652a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:60138/0 (socket says 172.21.15.117:60138) 2024-05-15T12:43:11.008 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.976+0000 7f6d47fff640 1 -- 172.21.15.117:0/3188308714 learned_addr learned my addr 172.21.15.117:0/3188308714 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:43:11.008 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.976+0000 7f6d47fff640 10 monclient(hunting): get_auth_request con 0x7f6d400cfab0 auth_method 0 2024-05-15T12:43:11.008 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.976+0000 7f6d47fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:11.008 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.976+0000 7f6d47fff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:11.008 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.976+0000 7f6d47fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:11.009 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.977+0000 7f6d47fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:11.009 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.977+0000 7f6d47fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:11.009 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.977+0000 7f6d47fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:11.009 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.978+0000 7f99417fa640 1 -- 172.21.15.117:0/363281595 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f9938011de0 con 0x7f99440716a0 2024-05-15T12:43:11.009 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.978+0000 7f99417fa640 10 client.4318.objecter ms_dispatch 0x7f9944000b90 osd_map(26..26 src has 1..26) v4 2024-05-15T12:43:11.009 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.978+0000 7f99417fa640 3 client.4318.objecter handle_osd_map got epochs [26,26] > 0 2024-05-15T12:43:11.009 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.978+0000 7f99417fa640 3 client.4318.objecter handle_osd_map decoding full epoch 26 2024-05-15T12:43:11.010 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.978+0000 7f99417fa640 10 client.4318.objecter checking command 1 2024-05-15T12:43:11.010 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.978+0000 7f99417fa640 1 --2- 172.21.15.117:0/363281595 >> [v2:172.21.15.117:6826/1673738625,v1:172.21.15.117:6827/1673738625] conn(0x7f99300732f0 0x7f9930075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.010 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.978+0000 7f99417fa640 20 client.4318.objecter _get_session s=0x7f9930072c20 osd=0 3 2024-05-15T12:43:11.010 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.978+0000 7f99417fa640 20 client.4318.objecter put_session s=0x7f9930072c20 osd=0 3 2024-05-15T12:43:11.010 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.978+0000 7f99417fa640 15 client.4318.objecter _session_command_op_remove -1 1 2024-05-15T12:43:11.010 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.978+0000 7f99417fa640 20 client.4318.objecter _get_session s=0x7f9930072c20 osd=0 3 2024-05-15T12:43:11.010 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.978+0000 7f99417fa640 20 client.4318.objecter get_session s=0x7f9930072c20 osd=0 3 2024-05-15T12:43:11.010 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.978+0000 7f99417fa640 15 client.4318.objecter _session_command_op_assign 0 1 2024-05-15T12:43:11.010 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.978+0000 7f99417fa640 20 client.4318.objecter put_session s=0x7f9930072c20 osd=0 4 2024-05-15T12:43:11.010 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.978+0000 7f99417fa640 10 client.4318.objecter _send_command 1 2024-05-15T12:43:11.010 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.978+0000 7f99417fa640 1 -- 172.21.15.117:0/363281595 --> [v2:172.21.15.117:6826/1673738625,v1:172.21.15.117:6827/1673738625] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9938006780 con 0x7f99300732f0 2024-05-15T12:43:11.010 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.978+0000 7f99417fa640 20 client.4318.objecter dump_active .. 0 homeless 2024-05-15T12:43:11.010 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.978+0000 7f99417fa640 1 -- 172.21.15.117:0/363281595 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9938055250 con 0x7f99440716a0 2024-05-15T12:43:11.010 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.978+0000 7f99417fa640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-05-15T12:43:11.011 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.978+0000 7f801ca06640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:11.011 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.978+0000 7f801ca06640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:11.011 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.978+0000 7f801ca06640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:11.011 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.979+0000 7f9949473640 1 --2- 172.21.15.117:0/363281595 >> [v2:172.21.15.117:6826/1673738625,v1:172.21.15.117:6827/1673738625] conn(0x7f99300732f0 0x7f9930075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:11.011 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.979+0000 7f9949473640 10 monclient: get_auth_request con 0x7f99300732f0 auth_method 0 2024-05-15T12:43:11.013 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.980+0000 7f9e17fff640 1 -- 172.21.15.117:0/3422287661 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e10038070 con 0x7f9e20066ec0 2024-05-15T12:43:11.013 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.980+0000 7f9e17fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:11.013 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.980+0000 7f9e17fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:43:11.013 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.980+0000 7f9e17fff640 10 monclient: dump: 2024-05-15T12:43:11.013 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:11.013 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:11.013 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.013 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.013 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:11.013 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.014 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:11.014 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:11.014 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:11.014 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.014 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.980+0000 7f9e17fff640 10 monclient: _finish_auth 0 2024-05-15T12:43:11.014 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.980+0000 7f9e17fff640 10 monclient: _check_auth_tickets 2024-05-15T12:43:11.014 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.980+0000 7f9e17fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:11.014 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.980+0000 7f9e17fff640 1 -- 172.21.15.117:0/3422287661 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e10031740 con 0x7f9e20066ec0 2024-05-15T12:43:11.014 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.980+0000 7f9e17fff640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:11.014 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.980+0000 7f9e17fff640 1 -- 172.21.15.117:0/3422287661 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e1002f050 con 0x7f9e20066ec0 2024-05-15T12:43:11.014 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.980+0000 7f9e17fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:11.014 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.980+0000 7f9e17fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T12:43:11.014 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.980+0000 7f9e17fff640 10 monclient: dump: 2024-05-15T12:43:11.014 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:11.014 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:11.015 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.015 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.015 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:11.015 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.015 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:11.015 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:11.015 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:11.015 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.015 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.980+0000 7f9e267d8640 5 monclient: authenticate success, global_id 4274 2024-05-15T12:43:11.015 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.980+0000 7f9e267d8640 10 monclient: get_monmap_and_config success 2024-05-15T12:43:11.015 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.980+0000 7f9e267d8640 4 set_mon_vals no callback set 2024-05-15T12:43:11.015 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.981+0000 7f9e267d8640 10 monclient: shutdown 2024-05-15T12:43:11.015 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.981+0000 7f9e267d8640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:11.015 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.981+0000 7f9e267d8640 1 -- 172.21.15.117:0/3422287661 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9e20066ec0 msgr2=0x7f9e20145b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.015 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.981+0000 7f9e267d8640 1 --2- 172.21.15.117:0/3422287661 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9e20066ec0 0x7f9e20145b20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9e10009870 tx=0x7f9e1002b7e0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.016 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.981+0000 7f9e267d8640 1 -- 172.21.15.117:0/3422287661 shutdown_connections 2024-05-15T12:43:11.016 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.981+0000 7f9e267d8640 1 --2- 172.21.15.117:0/3422287661 >> v2:172.21.15.117:3300/0 conn(0x7f9e200adce0 0x7f9e20146060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.016 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.981+0000 7f9e267d8640 1 --2- 172.21.15.117:0/3422287661 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9e20066ec0 0x7f9e20145b20 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.016 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.981+0000 7f9e267d8640 1 -- 172.21.15.117:0/3422287661 >> 172.21.15.117:0/3422287661 conn(0x7f9e200b1640 msgr2=0x7f9e2006ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:11.016 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.981+0000 7f9e267d8640 1 -- 172.21.15.117:0/3422287661 shutdown_connections 2024-05-15T12:43:11.016 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.981+0000 7f9e267d8640 1 -- 172.21.15.117:0/3422287661 wait complete. 2024-05-15T12:43:11.016 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.982+0000 7f9e267d8640 10 monclient: build_initial_monmap 2024-05-15T12:43:11.016 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.982+0000 7f9e267d8640 10 monclient: monmap: 2024-05-15T12:43:11.016 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:11.016 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:43:11.016 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-05-15T12:43:11.016 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-05-15T12:43:11.016 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:11.016 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.016 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T12:43:11.016 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T12:43:11.017 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T12:43:11.017 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.017 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.982+0000 7f9e267d8640 1 librados: starting msgr at 2024-05-15T12:43:11.017 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.982+0000 7f9e267d8640 1 librados: starting objecter 2024-05-15T12:43:11.017 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.982+0000 7f9e267d8640 1 Processor -- start 2024-05-15T12:43:11.017 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.982+0000 7f9e267d8640 1 -- start start 2024-05-15T12:43:11.017 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.982+0000 7f9e267d8640 1 librados: setting wanted keys 2024-05-15T12:43:11.017 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.982+0000 7f9e267d8640 1 librados: calling monclient init 2024-05-15T12:43:11.017 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.982+0000 7f9e267d8640 10 monclient: init 2024-05-15T12:43:11.018 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.982+0000 7f091cff9640 1 -- 172.21.15.117:0/2876865113 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0914015020 con 0x7f09200b4d00 2024-05-15T12:43:11.018 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.982+0000 7f091cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:11.018 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.982+0000 7f091cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:43:11.018 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.982+0000 7f091cff9640 10 monclient: dump: 2024-05-15T12:43:11.018 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:11.018 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:11.018 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.019 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.019 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:11.019 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.019 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:11.019 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:11.019 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:11.019 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.019 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.982+0000 7f091cff9640 10 monclient: _finish_auth 0 2024-05-15T12:43:11.019 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.982+0000 7f091cff9640 10 monclient: _check_auth_tickets 2024-05-15T12:43:11.019 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.982+0000 7f091cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:11.019 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.982+0000 7f091cff9640 1 -- 172.21.15.117:0/2876865113 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09140118b0 con 0x7f09200b4d00 2024-05-15T12:43:11.019 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.982+0000 7f09266ef640 5 monclient: authenticate success, global_id 4283 2024-05-15T12:43:11.019 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.982+0000 7f091cff9640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:11.019 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.982+0000 7f09266ef640 10 monclient: _renew_subs 2024-05-15T12:43:11.019 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.982+0000 7f091cff9640 1 -- 172.21.15.117:0/2876865113 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0914011ca0 con 0x7f09200b4d00 2024-05-15T12:43:11.019 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.982+0000 7f09266ef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T12:43:11.020 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.982+0000 7f09266ef640 1 -- 172.21.15.117:0/2876865113 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09200ac9b0 con 0x7f09200b4d00 2024-05-15T12:43:11.020 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.982+0000 7f091cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:11.020 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.982+0000 7f091e7fc640 4 set_mon_vals no callback set 2024-05-15T12:43:11.020 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.982+0000 7f091cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T12:43:11.020 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.982+0000 7f091cff9640 10 monclient: dump: 2024-05-15T12:43:11.020 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:11.020 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:11.020 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.020 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.020 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:11.020 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.020 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:11.020 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:11.020 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:11.020 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.020 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.983+0000 7f09266ef640 10 client.4283.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:43:11.020 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.983+0000 7f09266ef640 10 monclient: _renew_subs 2024-05-15T12:43:11.021 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.983+0000 7f09266ef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T12:43:11.021 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.983+0000 7f09266ef640 1 -- 172.21.15.117:0/2876865113 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f092013ea70 con 0x7f09200b4d00 2024-05-15T12:43:11.021 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.983+0000 7f09266ef640 1 librados: init done 2024-05-15T12:43:11.021 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.983+0000 7f6d46ffd640 1 --2- 172.21.15.117:0/3188308714 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d40033560 0x7f6d40164d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:11.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.983+0000 7f38367fc640 1 -- 172.21.15.117:0/1662615750 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3838019070 con 0x7f384013dac0 2024-05-15T12:43:11.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.983+0000 7f38367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:11.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.983+0000 7f38367fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:43:11.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.983+0000 7f38367fc640 10 monclient: dump: 2024-05-15T12:43:11.022 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:11.022 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:11.022 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.023 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.023 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:11.023 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.023 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:11.023 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:11.023 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:11.023 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.023 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.983+0000 7f38367fc640 10 monclient: _finish_auth 0 2024-05-15T12:43:11.023 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.983+0000 7f38367fc640 10 monclient: _check_auth_tickets 2024-05-15T12:43:11.023 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.983+0000 7f38367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:11.023 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.983+0000 7f38367fc640 1 -- 172.21.15.117:0/1662615750 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f383800cb20 con 0x7f384013dac0 2024-05-15T12:43:11.023 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.983+0000 7f3847cbe640 5 monclient: authenticate success, global_id 4280 2024-05-15T12:43:11.023 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.983+0000 7f3847cbe640 10 monclient: _renew_subs 2024-05-15T12:43:11.023 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.983+0000 7f3847cbe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T12:43:11.023 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.983+0000 7f3847cbe640 1 -- 172.21.15.117:0/1662615750 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f384006e680 con 0x7f384013dac0 2024-05-15T12:43:11.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.983+0000 7f9e267d8640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:11.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.983+0000 7f3847cbe640 10 client.4280.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:43:11.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.983+0000 7f38367fc640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:11.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.983+0000 7f9e267d8640 10 monclient: _add_conns ranks=[0,2,1] 2024-05-15T12:43:11.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.983+0000 7f9e267d8640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e20066ae0 0x7f9e20147490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.983+0000 7f9e267d8640 10 monclient(hunting): picked mon.noname-a con 0x7f9e20066ae0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T12:43:11.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.983+0000 7f9e267d8640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9e200adce0 0x7f9e20147df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.983+0000 7f3837fff640 4 set_mon_vals no callback set 2024-05-15T12:43:11.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.983+0000 7f38367fc640 1 -- 172.21.15.117:0/1662615750 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3838014440 con 0x7f384013dac0 2024-05-15T12:43:11.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.983+0000 7f38367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:11.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.983+0000 7f38367fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T12:43:11.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.983+0000 7f38367fc640 10 monclient: dump: 2024-05-15T12:43:11.026 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:11.026 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:11.026 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.026 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.026 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:11.026 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.027 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:11.027 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:11.027 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:11.027 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.983+0000 7f3847cbe640 10 monclient: _renew_subs 2024-05-15T12:43:11.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.983+0000 7f3847cbe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T12:43:11.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.983+0000 7f3847cbe640 1 -- 172.21.15.117:0/1662615750 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f384006ec40 con 0x7f384013dac0 2024-05-15T12:43:11.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.983+0000 7f3847cbe640 1 librados: init done 2024-05-15T12:43:11.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.983+0000 7f801d207640 1 --2- 172.21.15.117:0/3984485137 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8010032db0 0x7f80100ce410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:11.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.983+0000 7f801ca06640 10 monclient(hunting): handle_auth_done global_id 4296 payload 824 2024-05-15T12:43:11.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.983+0000 7f801ca06640 1 -- 172.21.15.117:0/3984485137 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8010032db0 msgr2=0x7f80100ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.983+0000 7f801ca06640 1 --2- 172.21.15.117:0/3984485137 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8010032db0 0x7f80100ce410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.983+0000 7f801ca06640 1 -- 172.21.15.117:0/3984485137 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8010032970 msgr2=0x7f80100c59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.983+0000 7f801ca06640 1 --2- 172.21.15.117:0/3984485137 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8010032970 0x7f80100c59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.983+0000 7f801ca06640 10 monclient: _finish_hunting 0 2024-05-15T12:43:11.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.983+0000 7f801ca06640 1 monclient: found mon.noname-a 2024-05-15T12:43:11.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.983+0000 7f801ca06640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:11.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.983+0000 7f801ca06640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.117:3300/0 2024-05-15T12:43:11.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.983+0000 7f801ca06640 1 -- 172.21.15.117:0/3984485137 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80100c51a0 con 0x7f801002e200 2024-05-15T12:43:11.029 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.983+0000 7f9e267d8640 10 monclient(hunting): picked mon.noname-c con 0x7f9e200adce0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:43:11.029 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.983+0000 7f9e267d8640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9e20149ca0 0x7f9e20148330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.029 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.983+0000 7f9e267d8640 10 monclient(hunting): picked mon.noname-b con 0x7f9e20149ca0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:43:11.029 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.983+0000 7f9e267d8640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:11.029 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.983+0000 7f9e267d8640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e20146720 con 0x7f9e20066ae0 2024-05-15T12:43:11.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.984+0000 7f09266ef640 10 client.4283.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:11.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.984+0000 7f09266ef640 15 client.4283.objecter _session_command_op_assign -1 1 2024-05-15T12:43:11.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.984+0000 7f09266ef640 20 client.4283.objecter _get_session osd=-1 returning homeless 2024-05-15T12:43:11.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.984+0000 7f09266ef640 10 client.4283.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:43:11.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.983+0000 7f9e267d8640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:11.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.984+0000 7f09266ef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T12:43:11.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.984+0000 7f09266ef640 1 -- 172.21.15.117:0/2876865113 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f092004f0e0 con 0x7f09200b4d00 2024-05-15T12:43:11.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.983+0000 7f9e267d8640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e201465a0 con 0x7f9e20149ca0 2024-05-15T12:43:11.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.983+0000 7f9e267d8640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:11.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.983+0000 7f9e267d8640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2014ceb0 con 0x7f9e200adce0 2024-05-15T12:43:11.032 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.983+0000 7f9e267d8640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:11.032 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.983+0000 7f9e267d8640 10 monclient(hunting): authenticate will time out at 1354.225223s 2024-05-15T12:43:11.032 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.984+0000 7f3847cbe640 10 client.4280.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:11.032 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.984+0000 7f3847cbe640 15 client.4280.objecter _session_command_op_assign -1 1 2024-05-15T12:43:11.032 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.984+0000 7f3847cbe640 20 client.4280.objecter _get_session osd=-1 returning homeless 2024-05-15T12:43:11.032 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.984+0000 7f3847cbe640 10 client.4280.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:43:11.032 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.984+0000 7f3847cbe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T12:43:11.032 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.984+0000 7f3847cbe640 1 -- 172.21.15.117:0/1662615750 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f384004f0e0 con 0x7f384013dac0 2024-05-15T12:43:11.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.985+0000 7f99417fa640 10 client.4318.objecter ms_handle_connect 0x7f99300732f0 2024-05-15T12:43:11.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.985+0000 7f9949473640 1 --2- 172.21.15.117:0/363281595 >> [v2:172.21.15.117:6826/1673738625,v1:172.21.15.117:6827/1673738625] conn(0x7f99300732f0 0x7f9930075710 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:11.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.985+0000 7f6d46ffd640 10 monclient(hunting): get_auth_request con 0x7f6d40033560 auth_method 0 2024-05-15T12:43:11.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.986+0000 7f6d46ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:11.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.986+0000 7f6d46ffd640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:11.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.986+0000 7f6d46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:11.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.986+0000 7f07dd07f640 1 --2- 172.21.15.117:0/1548430383 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f07c4003a00 0x7f07c4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:11.035 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.987+0000 7f6d47fff640 10 monclient(hunting): handle_auth_done global_id 4293 payload 824 2024-05-15T12:43:11.035 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.987+0000 7f6d47fff640 1 -- 172.21.15.117:0/3188308714 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d40033560 msgr2=0x7f6d40164d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.035 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.987+0000 7f6d47fff640 1 --2- 172.21.15.117:0/3188308714 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d40033560 0x7f6d40164d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.035 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.987+0000 7f6d47fff640 1 -- 172.21.15.117:0/3188308714 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6d400071d0 msgr2=0x7f6d400c1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.035 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.987+0000 7f6d47fff640 1 --2- 172.21.15.117:0/3188308714 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6d400071d0 0x7f6d400c1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.035 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.987+0000 7f6d47fff640 10 monclient: _finish_hunting 0 2024-05-15T12:43:11.035 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.987+0000 7f6d47fff640 1 monclient: found mon.noname-a 2024-05-15T12:43:11.035 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.987+0000 7f6d47fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:11.035 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.987+0000 7f6d47fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.117:3300/0 2024-05-15T12:43:11.036 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.987+0000 7f6d47fff640 1 -- 172.21.15.117:0/3188308714 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d400c3060 con 0x7f6d400cfab0 2024-05-15T12:43:11.036 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.989+0000 7f07dd07f640 10 monclient(hunting): get_auth_request con 0x7f07c4003a00 auth_method 0 2024-05-15T12:43:11.036 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.990+0000 7f07dd07f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:11.036 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.990+0000 7f07dd07f640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:11.036 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.990+0000 7f07dd07f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-05-15T12:43:11.036 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.992+0000 7f6d47fff640 1 --2- 172.21.15.117:0/3188308714 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6d400cfab0 0x7f6d401652a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6d4806a9b0 tx=0x7f6d4806a0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:11.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.992+0000 7f6d33fff640 10 client.?.objecter ms_handle_connect 0x7f6d400cfab0 2024-05-15T12:43:11.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.992+0000 7f6d33fff640 10 client.?.objecter resend_mon_ops 2024-05-15T12:43:11.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.993+0000 7f9e25fd7640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9e20149ca0 0x7f9e20148330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:11.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.995+0000 7f9e25fd7640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9e20149ca0 0x7f9e20148330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:42460/0 (socket says 172.21.15.117:42460) 2024-05-15T12:43:11.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.995+0000 7f9e25fd7640 1 -- 172.21.15.117:0/2577752866 learned_addr learned my addr 172.21.15.117:0/2577752866 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:43:11.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.995+0000 7f9e25fd7640 10 monclient(hunting): get_auth_request con 0x7f9e20149ca0 auth_method 0 2024-05-15T12:43:11.038 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.995+0000 7f9e25fd7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:11.038 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.995+0000 7f9e25fd7640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:11.038 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.995+0000 7f9e25fd7640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:11.038 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.996+0000 7f07dd880640 1 --2- 172.21.15.117:0/1548430383 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f07c4005210 0x7f07c400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:11.038 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.996+0000 7f07dd880640 10 monclient(hunting): get_auth_request con 0x7f07c4005210 auth_method 0 2024-05-15T12:43:11.038 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.996+0000 7f07dd880640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:11.038 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.996+0000 7f07dd880640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:11.039 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.996+0000 7f07dd880640 10 monclient(hunting): _init_auth already have auth, reseting 2024-05-15T12:43:11.039 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.997+0000 7f9e25fd7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:11.039 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.997+0000 7f9e25fd7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:11.039 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.997+0000 7f9e25fd7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:11.039 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.999+0000 7f801ca06640 1 --2- 172.21.15.117:0/3984485137 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f801002e200 0x7f80100a3c80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f801807adc0 tx=0x7f8018068a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:11.039 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:10.999+0000 7f80157fa640 10 client.?.objecter ms_handle_connect 0x7f801002e200 2024-05-15T12:43:11.040 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.000+0000 7f80157fa640 10 client.?.objecter resend_mon_ops 2024-05-15T12:43:11.040 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.000+0000 7f07dc87e640 1 --2- 172.21.15.117:0/1548430383 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f07c4004800 0x7f07c4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:11.040 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.000+0000 7f9e25fd7640 10 monclient(hunting): handle_auth_done global_id 4339 payload 824 2024-05-15T12:43:11.040 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.000+0000 7f9e25fd7640 1 -- 172.21.15.117:0/2577752866 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9e200adce0 msgr2=0x7f9e20147df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.040 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.000+0000 7f9e25fd7640 1 --2- 172.21.15.117:0/2577752866 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9e200adce0 0x7f9e20147df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.040 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.000+0000 7f9e25fd7640 1 -- 172.21.15.117:0/2577752866 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e20066ae0 msgr2=0x7f9e20147490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.041 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.000+0000 7f9e25fd7640 1 --2- 172.21.15.117:0/2577752866 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e20066ae0 0x7f9e20147490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.041 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.000+0000 7f9e25fd7640 10 monclient: _finish_hunting 0 2024-05-15T12:43:11.041 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.000+0000 7f9e25fd7640 1 monclient: found mon.noname-b 2024-05-15T12:43:11.041 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.000+0000 7f9e25fd7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:11.041 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.000+0000 7f9e25fd7640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2024-05-15T12:43:11.041 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.000+0000 7f9e25fd7640 1 -- 172.21.15.117:0/2577752866 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e201495a0 con 0x7f9e20149ca0 2024-05-15T12:43:11.041 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.000+0000 7f07dd880640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:11.041 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.000+0000 7f07dd880640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:11.041 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.000+0000 7f07dd880640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-05-15T12:43:11.042 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.001+0000 7f07dc87e640 10 monclient(hunting): get_auth_request con 0x7f07c4004800 auth_method 0 2024-05-15T12:43:11.042 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.001+0000 7f07dc87e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:11.042 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.001+0000 7f07dc87e640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:11.042 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.001+0000 7f07dc87e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-05-15T12:43:11.044 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.001+0000 7f6d33fff640 1 -- 172.21.15.117:0/3188308714 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d4804ffd0 con 0x7f6d400cfab0 2024-05-15T12:43:11.044 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.001+0000 7f6d33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:11.044 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.001+0000 7f6d33fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T12:43:11.044 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.001+0000 7f6d33fff640 10 monclient: dump: 2024-05-15T12:43:11.044 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:11.044 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:11.044 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.044 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.044 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:11.044 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.044 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:11.045 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:11.045 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:11.045 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.045 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.001+0000 7f6d33fff640 10 monclient: _finish_auth 0 2024-05-15T12:43:11.045 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.001+0000 7f6d33fff640 10 monclient: _check_auth_tickets 2024-05-15T12:43:11.045 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.001+0000 7f6d33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:11.045 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.001+0000 7f6d33fff640 1 -- 172.21.15.117:0/3188308714 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d48090020 con 0x7f6d400cfab0 2024-05-15T12:43:11.045 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.001+0000 7f6d33fff640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:11.045 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.001+0000 7f6d33fff640 1 -- 172.21.15.117:0/3188308714 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d480733d0 con 0x7f6d400cfab0 2024-05-15T12:43:11.045 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.001+0000 7f6d33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:11.045 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.001+0000 7f6d33fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T12:43:11.045 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.001+0000 7f6d33fff640 10 monclient: dump: 2024-05-15T12:43:11.045 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:11.045 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:11.046 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.046 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.046 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:11.046 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.046 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:11.046 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:11.046 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:11.046 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.046 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.001+0000 7f6d4c9d3640 5 monclient: authenticate success, global_id 4293 2024-05-15T12:43:11.046 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.001+0000 7f6d4c9d3640 10 monclient: _renew_subs 2024-05-15T12:43:11.046 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.001+0000 7f6d4c9d3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:43:11.046 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.001+0000 7f6d4c9d3640 1 -- 172.21.15.117:0/3188308714 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d400c35b0 con 0x7f6d400cfab0 2024-05-15T12:43:11.046 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.001+0000 7f6d45ffb640 4 set_mon_vals no callback set 2024-05-15T12:43:11.047 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.001+0000 7f6d4c9d3640 10 client.4293.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:43:11.047 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.001+0000 7f6d4c9d3640 10 monclient: _renew_subs 2024-05-15T12:43:11.047 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.001+0000 7f6d4c9d3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:43:11.047 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.001+0000 7f6d4c9d3640 1 -- 172.21.15.117:0/3188308714 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d40030320 con 0x7f6d400cfab0 2024-05-15T12:43:11.047 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.001+0000 7f6d4c9d3640 1 librados: init done 2024-05-15T12:43:11.047 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.002+0000 7f6d4c9d3640 10 client.4293.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:11.047 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.002+0000 7f6d4c9d3640 15 client.4293.objecter _session_command_op_assign -1 1 2024-05-15T12:43:11.047 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.002+0000 7f6d4c9d3640 20 client.4293.objecter _get_session osd=-1 returning homeless 2024-05-15T12:43:11.047 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.002+0000 7f6d4c9d3640 10 client.4293.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:43:11.047 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.002+0000 7f6d4c9d3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:43:11.047 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.002+0000 7f6d4c9d3640 1 -- 172.21.15.117:0/3188308714 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6d40008080 con 0x7f6d400cfab0 2024-05-15T12:43:11.048 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.004+0000 7f07dd880640 10 monclient(hunting): handle_auth_done global_id 4287 payload 293 2024-05-15T12:43:11.048 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.004+0000 7f07dd880640 1 -- 172.21.15.117:0/1548430383 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f07c4003a00 msgr2=0x7f07c4003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:43:11.048 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.004+0000 7f07dd880640 1 --2- 172.21.15.117:0/1548430383 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f07c4003a00 0x7f07c4003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.048 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.004+0000 7f07dd880640 1 -- 172.21.15.117:0/1548430383 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f07c4004800 msgr2=0x7f07c4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:43:11.048 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.004+0000 7f07dd880640 1 --2- 172.21.15.117:0/1548430383 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f07c4004800 0x7f07c4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.048 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.004+0000 7f07dd880640 10 monclient: _finish_hunting 0 2024-05-15T12:43:11.048 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.004+0000 7f07dd880640 1 monclient: found mon.b 2024-05-15T12:43:11.048 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.004+0000 7f07dd880640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:11.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.004+0000 7f07dd880640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:43:11.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.004+0000 7f07dd880640 1 -- 172.21.15.117:0/1548430383 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07c4003520 con 0x7f07c4005210 2024-05-15T12:43:11.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.004+0000 7f07dd880640 10 monclient: _finish_auth 0 2024-05-15T12:43:11.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.004+0000 7f07dd880640 10 monclient: _check_auth_tickets 2024-05-15T12:43:11.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.004+0000 7f07dd880640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:11.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.004+0000 7f07df2e3640 5 monclient: authenticate success, global_id 4287 2024-05-15T12:43:11.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.004+0000 7f07df2e3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:43:11.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.005+0000 7f07dd880640 1 -- 172.21.15.117:0/1548430383 reap_dead start 2024-05-15T12:43:11.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.006+0000 7f9e25fd7640 1 --2- 172.21.15.117:0/2577752866 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9e20149ca0 0x7f9e20148330 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9e1800b9a0 tx=0x7f9e1800be60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:11.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.006+0000 7f9e15ffb640 10 client.?.objecter ms_handle_connect 0x7f9e20149ca0 2024-05-15T12:43:11.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.006+0000 7f9e15ffb640 10 client.?.objecter resend_mon_ops 2024-05-15T12:43:11.050 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.009+0000 7f07dd880640 1 --2- 172.21.15.117:0/1548430383 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f07c4005210 0x7f07c400b9c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f07d000a7b0 tx=0x7f07c4005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d057f80c3d5f3daa server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:11.051 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.011+0000 7f9e15ffb640 1 -- 172.21.15.117:0/2577752866 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e18019070 con 0x7f9e20149ca0 2024-05-15T12:43:11.051 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.011+0000 7f9e15ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:11.051 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.011+0000 7f9e15ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:43:11.051 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.011+0000 7f9e15ffb640 10 monclient: dump: 2024-05-15T12:43:11.051 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:11.051 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:11.051 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.051 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.051 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:11.052 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.052 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:11.052 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:11.052 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:11.052 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.052 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.011+0000 7f9e15ffb640 10 monclient: _finish_auth 0 2024-05-15T12:43:11.052 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.011+0000 7f9e15ffb640 10 monclient: _check_auth_tickets 2024-05-15T12:43:11.052 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.011+0000 7f9e15ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:11.052 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.011+0000 7f9e15ffb640 1 -- 172.21.15.117:0/2577752866 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e1800cbc0 con 0x7f9e20149ca0 2024-05-15T12:43:11.052 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.011+0000 7f9e15ffb640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:11.052 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.011+0000 7f9e15ffb640 1 -- 172.21.15.117:0/2577752866 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e18014440 con 0x7f9e20149ca0 2024-05-15T12:43:11.052 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.011+0000 7f9e15ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:11.052 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.011+0000 7f9e15ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:43:11.052 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.011+0000 7f9e15ffb640 10 monclient: dump: 2024-05-15T12:43:11.052 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:11.052 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:11.053 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.053 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.053 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:11.053 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.053 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:11.053 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:11.053 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:11.053 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.053 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.011+0000 7f9e267d8640 5 monclient: authenticate success, global_id 4339 2024-05-15T12:43:11.053 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.011+0000 7f9e267d8640 10 monclient: _renew_subs 2024-05-15T12:43:11.053 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.011+0000 7f9e267d8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:43:11.053 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.011+0000 7f9e267d8640 1 -- 172.21.15.117:0/2577752866 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e20149880 con 0x7f9e20149ca0 2024-05-15T12:43:11.053 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.011+0000 7f9e177fe640 4 set_mon_vals no callback set 2024-05-15T12:43:11.053 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.011+0000 7f9e267d8640 10 client.4339.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:43:11.053 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.011+0000 7f9e267d8640 10 monclient: _renew_subs 2024-05-15T12:43:11.054 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.011+0000 7f9e267d8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:43:11.054 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.011+0000 7f9e267d8640 1 -- 172.21.15.117:0/2577752866 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e2013a620 con 0x7f9e20149ca0 2024-05-15T12:43:11.054 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.011+0000 7f9e267d8640 1 librados: init done 2024-05-15T12:43:11.054 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.012+0000 7f9e267d8640 10 client.4339.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:11.054 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.012+0000 7f9e267d8640 15 client.4339.objecter _session_command_op_assign -1 1 2024-05-15T12:43:11.054 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.012+0000 7f9e267d8640 20 client.4339.objecter _get_session osd=-1 returning homeless 2024-05-15T12:43:11.054 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.012+0000 7f9e267d8640 10 client.4339.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:43:11.054 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.012+0000 7f9e267d8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:43:11.054 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.012+0000 7f9e267d8640 1 -- 172.21.15.117:0/2577752866 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9e200764a0 con 0x7f9e20149ca0 2024-05-15T12:43:11.057 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.021+0000 7f07cffff640 1 -- 172.21.15.117:0/1548430383 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07d000a3d0 con 0x7f07c4005210 2024-05-15T12:43:11.057 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.021+0000 7f07cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:11.057 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.021+0000 7f07cffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:43:11.057 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.021+0000 7f07cffff640 10 monclient: dump: 2024-05-15T12:43:11.058 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:11.058 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:11.058 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.058 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.058 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:11.058 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.058 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:11.058 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:11.058 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:11.058 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.058 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.021+0000 7f07cffff640 1 -- 172.21.15.117:0/1548430383 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07d0034670 con 0x7f07c4005210 2024-05-15T12:43:11.058 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.021+0000 7f07cffff640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:11.058 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.021+0000 7f07cffff640 1 -- 172.21.15.117:0/1548430383 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07d0034a60 con 0x7f07c4005210 2024-05-15T12:43:11.059 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.021+0000 7f07cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:11.059 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.021+0000 7f07cffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:43:11.059 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.021+0000 7f07cffff640 10 monclient: dump: 2024-05-15T12:43:11.059 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:11.059 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:11.059 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.059 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.059 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:11.059 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.059 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:11.059 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:11.059 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:11.059 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.059 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.021+0000 7f07df2e3640 10 monclient: get_monmap_and_config success 2024-05-15T12:43:11.059 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.021+0000 7f07df2e3640 4 set_mon_vals no callback set 2024-05-15T12:43:11.059 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.022+0000 7f07df2e3640 10 monclient: shutdown 2024-05-15T12:43:11.060 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.022+0000 7f07df2e3640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:11.060 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.022+0000 7f07df2e3640 1 -- 172.21.15.117:0/1548430383 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f07c4005210 msgr2=0x7f07c400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.060 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.022+0000 7f07df2e3640 1 --2- 172.21.15.117:0/1548430383 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f07c4005210 0x7f07c400b9c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f07d000a7b0 tx=0x7f07c4005830 comp rx=0 tx=0).stop 2024-05-15T12:43:11.060 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.022+0000 7f07df2e3640 1 -- 172.21.15.117:0/1548430383 shutdown_connections 2024-05-15T12:43:11.060 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.022+0000 7f07df2e3640 1 --2- 172.21.15.117:0/1548430383 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f07c4005210 0x7f07c400b9c0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.060 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.022+0000 7f07df2e3640 1 -- 172.21.15.117:0/1548430383 >> 172.21.15.117:0/1548430383 conn(0x7f07d8066cc0 msgr2=0x7f07d80670d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:11.060 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.022+0000 7f07df2e3640 1 -- 172.21.15.117:0/1548430383 shutdown_connections 2024-05-15T12:43:11.060 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.022+0000 7f07df2e3640 1 -- 172.21.15.117:0/1548430383 wait complete. 2024-05-15T12:43:11.060 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.023+0000 7f07df2e3640 10 monclient: build_initial_monmap 2024-05-15T12:43:11.060 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.023+0000 7f07df2e3640 10 monclient: monmap: 2024-05-15T12:43:11.060 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:11.060 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:43:11.060 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-05-15T12:43:11.060 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-05-15T12:43:11.060 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:11.060 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.061 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T12:43:11.061 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T12:43:11.061 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T12:43:11.061 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.061 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.024+0000 7f07df2e3640 1 librados: starting msgr at 2024-05-15T12:43:11.061 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.024+0000 7f07df2e3640 1 librados: starting objecter 2024-05-15T12:43:11.061 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.024+0000 7f07df2e3640 1 Processor -- start 2024-05-15T12:43:11.061 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.024+0000 7f07df2e3640 1 -- start start 2024-05-15T12:43:11.061 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.024+0000 7f07df2e3640 1 librados: setting wanted keys 2024-05-15T12:43:11.061 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.024+0000 7f07df2e3640 1 librados: calling monclient init 2024-05-15T12:43:11.061 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.024+0000 7f07df2e3640 10 monclient: init 2024-05-15T12:43:11.061 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.025+0000 7f07df2e3640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:11.061 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.025+0000 7f07df2e3640 10 monclient: _add_conns ranks=[2,0,1] 2024-05-15T12:43:11.061 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.025+0000 7f07df2e3640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f07c4005210 0x7f07d80af3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.061 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.025+0000 7f07df2e3640 10 monclient(hunting): picked mon.noname-c con 0x7f07c4005210 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:43:11.061 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.025+0000 7f07df2e3640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f07d80b0190 0x7f07d81db3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.062 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.026+0000 7f07df2e3640 10 monclient(hunting): picked mon.noname-a con 0x7f07d80b0190 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T12:43:11.062 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.026+0000 7f07df2e3640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f07d81db8e0 0x7f07d81d7830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.062 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.026+0000 7f07df2e3640 10 monclient(hunting): picked mon.noname-b con 0x7f07d81db8e0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:43:11.062 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.026+0000 7f07df2e3640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:11.062 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.026+0000 7f07df2e3640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07d80b2210 con 0x7f07d80b0190 2024-05-15T12:43:11.062 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.026+0000 7f07df2e3640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:11.062 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.026+0000 7f07df2e3640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07d80b2090 con 0x7f07d81db8e0 2024-05-15T12:43:11.062 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.026+0000 7f07df2e3640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:11.062 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.026+0000 7f07df2e3640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07d8145c20 con 0x7f07c4005210 2024-05-15T12:43:11.062 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.026+0000 7f07df2e3640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:11.062 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.026+0000 7f07df2e3640 10 monclient(hunting): authenticate will time out at 1354.268115s 2024-05-15T12:43:11.063 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.028+0000 7f656bfff640 1 -- 172.21.15.117:0/3888178903 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f6574018020 con 0x7f6578033590 2024-05-15T12:43:11.063 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.029+0000 7f656bfff640 1 --2- 172.21.15.117:0/3888178903 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f656406cee0 0x7f656406f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.063 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.029+0000 7f657f7fe640 1 --2- 172.21.15.117:0/3888178903 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f656406cee0 0x7f656406f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:11.063 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.029+0000 7f657f7fe640 10 monclient: get_auth_request con 0x7f656406cee0 auth_method 0 2024-05-15T12:43:11.063 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.029+0000 7f656bfff640 10 client.4277.objecter ms_handle_connect 0x7f656406cee0 2024-05-15T12:43:11.063 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.029+0000 7f657f7fe640 1 --2- 172.21.15.117:0/3888178903 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f656406cee0 0x7f656406f3a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6580068f90 tx=0x7f6580069630 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:11.064 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.033+0000 7f9e15ffb640 1 -- 172.21.15.117:0/2577752866 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f9e1801c070 con 0x7f9e20149ca0 2024-05-15T12:43:11.064 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.034+0000 7f9e15ffb640 1 --2- 172.21.15.117:0/2577752866 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f9e0406ceb0 0x7f9e0406f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.064 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.038+0000 7f9e15ffb640 1 -- 172.21.15.117:0/2577752866 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f9e18056e60 con 0x7f9e20149ca0 2024-05-15T12:43:11.064 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.038+0000 7f9e15ffb640 10 client.4339.objecter ms_dispatch 0x7f9e20000b90 osd_map(26..26 src has 1..26) v4 2024-05-15T12:43:11.064 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.038+0000 7f9e15ffb640 3 client.4339.objecter handle_osd_map got epochs [26,26] > 0 2024-05-15T12:43:11.064 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.038+0000 7f9e15ffb640 3 client.4339.objecter handle_osd_map decoding full epoch 26 2024-05-15T12:43:11.065 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.038+0000 7f9e15ffb640 10 client.4339.objecter checking command 1 2024-05-15T12:43:11.065 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.038+0000 7f9e15ffb640 1 --2- 172.21.15.117:0/2577752866 >> [v2:172.21.15.178:6824/248472851,v1:172.21.15.178:6825/248472851] conn(0x7f9e040732f0 0x7f9e04075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.065 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.038+0000 7f9e15ffb640 20 client.4339.objecter _get_session s=0x7f9e04072c20 osd=4 3 2024-05-15T12:43:11.065 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.038+0000 7f9e15ffb640 20 client.4339.objecter put_session s=0x7f9e04072c20 osd=4 3 2024-05-15T12:43:11.065 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.038+0000 7f9e15ffb640 15 client.4339.objecter _session_command_op_remove -1 1 2024-05-15T12:43:11.065 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.038+0000 7f9e15ffb640 20 client.4339.objecter _get_session s=0x7f9e04072c20 osd=4 3 2024-05-15T12:43:11.065 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.038+0000 7f9e15ffb640 20 client.4339.objecter get_session s=0x7f9e04072c20 osd=4 3 2024-05-15T12:43:11.065 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.038+0000 7f9e15ffb640 15 client.4339.objecter _session_command_op_assign 4 1 2024-05-15T12:43:11.065 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.038+0000 7f9e15ffb640 20 client.4339.objecter put_session s=0x7f9e04072c20 osd=4 4 2024-05-15T12:43:11.065 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.038+0000 7f9e15ffb640 10 client.4339.objecter _send_command 1 2024-05-15T12:43:11.065 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.038+0000 7f9e15ffb640 1 -- 172.21.15.117:0/2577752866 --> [v2:172.21.15.178:6824/248472851,v1:172.21.15.178:6825/248472851] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9e18010e50 con 0x7f9e040732f0 2024-05-15T12:43:11.065 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.038+0000 7f9e15ffb640 20 client.4339.objecter dump_active .. 0 homeless 2024-05-15T12:43:11.066 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.038+0000 7f6d33fff640 1 -- 172.21.15.117:0/3188308714 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f6d480943c0 con 0x7f6d400cfab0 2024-05-15T12:43:11.066 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.039+0000 7f9e257d6640 1 --2- 172.21.15.117:0/2577752866 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f9e0406ceb0 0x7f9e0406f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:11.066 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.039+0000 7f6d33fff640 1 --2- 172.21.15.117:0/3188308714 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f6d2c06cea0 0x7f6d2c06f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.066 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.039+0000 7f9e257d6640 10 monclient: get_auth_request con 0x7f9e0406ceb0 auth_method 0 2024-05-15T12:43:11.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.039+0000 7f9e24fd5640 1 --2- 172.21.15.117:0/2577752866 >> [v2:172.21.15.178:6824/248472851,v1:172.21.15.178:6825/248472851] conn(0x7f9e040732f0 0x7f9e04075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:11.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.039+0000 7f6d477fe640 1 --2- 172.21.15.117:0/3188308714 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f6d2c06cea0 0x7f6d2c06f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:11.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.039+0000 7f6d477fe640 10 monclient: get_auth_request con 0x7f6d2c06cea0 auth_method 0 2024-05-15T12:43:11.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.039+0000 7f9e257d6640 1 --2- 172.21.15.117:0/2577752866 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f9e0406ceb0 0x7f9e0406f370 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c009650 tx=0x7f9e1c006ba0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:11.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.039+0000 7f9e15ffb640 10 client.4339.objecter ms_handle_connect 0x7f9e0406ceb0 2024-05-15T12:43:11.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.039+0000 7f9e24fd5640 10 monclient: get_auth_request con 0x7f9e040732f0 auth_method 0 2024-05-15T12:43:11.068 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.039+0000 7f07dd880640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f07d81db8e0 0x7f07d81d7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:11.068 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.040+0000 7f6d477fe640 1 --2- 172.21.15.117:0/3188308714 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f6d2c06cea0 0x7f6d2c06f360 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c036000 tx=0x7f6d3c030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:11.068 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.040+0000 7f6d33fff640 10 client.4293.objecter ms_handle_connect 0x7f6d2c06cea0 2024-05-15T12:43:11.069 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.040+0000 7f07dd880640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f07d81db8e0 0x7f07d81d7830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:42468/0 (socket says 172.21.15.117:42468) 2024-05-15T12:43:11.069 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.040+0000 7f07dd880640 1 -- 172.21.15.117:0/2001633136 learned_addr learned my addr 172.21.15.117:0/2001633136 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:43:11.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.040+0000 7f07dd880640 10 monclient(hunting): get_auth_request con 0x7f07d81db8e0 auth_method 0 2024-05-15T12:43:11.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.040+0000 7f07dd880640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:11.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.040+0000 7f07dd880640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:11.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.040+0000 7f07dd880640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:11.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.040+0000 7f07dd880640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:11.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.040+0000 7f07dd880640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:11.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.040+0000 7f07dd880640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:11.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.043+0000 7f07dd880640 10 monclient(hunting): handle_auth_done global_id 4342 payload 824 2024-05-15T12:43:11.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.043+0000 7f07dd880640 1 -- 172.21.15.117:0/2001633136 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f07c4005210 msgr2=0x7f07d80af3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.043+0000 7f07dd880640 1 --2- 172.21.15.117:0/2001633136 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f07c4005210 0x7f07d80af3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.043+0000 7f07dd880640 1 -- 172.21.15.117:0/2001633136 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f07d80b0190 msgr2=0x7f07d81db3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.043+0000 7f07dd880640 1 --2- 172.21.15.117:0/2001633136 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f07d80b0190 0x7f07d81db3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.043+0000 7f07dd880640 10 monclient: _finish_hunting 0 2024-05-15T12:43:11.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.043+0000 7f07dd880640 1 monclient: found mon.noname-b 2024-05-15T12:43:11.071 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.043+0000 7f07dd880640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:11.071 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.043+0000 7f07dd880640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2024-05-15T12:43:11.071 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.043+0000 7f07dd880640 1 -- 172.21.15.117:0/2001633136 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07d81205f0 con 0x7f07d81db8e0 2024-05-15T12:43:11.071 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.044+0000 7f9e15ffb640 10 client.4339.objecter ms_handle_connect 0x7f9e040732f0 2024-05-15T12:43:11.071 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.044+0000 7f9e24fd5640 1 --2- 172.21.15.117:0/2577752866 >> [v2:172.21.15.178:6824/248472851,v1:172.21.15.178:6825/248472851] conn(0x7f9e040732f0 0x7f9e04075710 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:11.074 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.046+0000 7f07dd880640 1 --2- 172.21.15.117:0/2001633136 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f07d81db8e0 0x7f07d81d7830 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f07d002cbc0 tx=0x7f07d80ab380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:11.074 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.046+0000 7f07cdffb640 10 client.?.objecter ms_handle_connect 0x7f07d81db8e0 2024-05-15T12:43:11.074 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.046+0000 7f07cdffb640 10 client.?.objecter resend_mon_ops 2024-05-15T12:43:11.074 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.048+0000 7f07cdffb640 1 -- 172.21.15.117:0/2001633136 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07d0043070 con 0x7f07d81db8e0 2024-05-15T12:43:11.074 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.048+0000 7f07cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:11.074 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.048+0000 7f07cdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:43:11.074 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.048+0000 7f07cdffb640 10 monclient: dump: 2024-05-15T12:43:11.074 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:11.074 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:11.074 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.075 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.075 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:11.075 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.075 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:11.075 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:11.075 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:11.075 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.048+0000 7f07cdffb640 10 monclient: _finish_auth 0 2024-05-15T12:43:11.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.048+0000 7f07cdffb640 10 monclient: _check_auth_tickets 2024-05-15T12:43:11.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.048+0000 7f07cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:11.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.048+0000 7f07df2e3640 5 monclient: authenticate success, global_id 4342 2024-05-15T12:43:11.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.048+0000 7f07df2e3640 10 monclient: _renew_subs 2024-05-15T12:43:11.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.048+0000 7f07df2e3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:43:11.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.049+0000 7f07df2e3640 1 -- 172.21.15.117:0/2001633136 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07d80af920 con 0x7f07d81db8e0 2024-05-15T12:43:11.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.049+0000 7f07df2e3640 10 client.4342.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:43:11.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.049+0000 7f07df2e3640 10 monclient: _renew_subs 2024-05-15T12:43:11.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.049+0000 7f07df2e3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:43:11.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.049+0000 7f07df2e3640 1 -- 172.21.15.117:0/2001633136 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07d80b05d0 con 0x7f07d81db8e0 2024-05-15T12:43:11.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.049+0000 7f07df2e3640 1 librados: init done 2024-05-15T12:43:11.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.050+0000 7f07df2e3640 10 client.4342.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:11.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.050+0000 7f07df2e3640 15 client.4342.objecter _session_command_op_assign -1 1 2024-05-15T12:43:11.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.050+0000 7f07df2e3640 20 client.4342.objecter _get_session osd=-1 returning homeless 2024-05-15T12:43:11.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.050+0000 7f07df2e3640 10 client.4342.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:43:11.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.050+0000 7f07df2e3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:43:11.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.050+0000 7f07df2e3640 1 -- 172.21.15.117:0/2001633136 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f07d80acb40 con 0x7f07d81db8e0 2024-05-15T12:43:11.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.051+0000 7f07cdffb640 1 -- 172.21.15.117:0/2001633136 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07d0004b60 con 0x7f07d81db8e0 2024-05-15T12:43:11.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.051+0000 7f07cdffb640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:11.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.051+0000 7f07cdffb640 1 -- 172.21.15.117:0/2001633136 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07d0039040 con 0x7f07d81db8e0 2024-05-15T12:43:11.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.051+0000 7f07cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:11.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.051+0000 7f07cdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:43:11.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.051+0000 7f07cdffb640 10 monclient: dump: 2024-05-15T12:43:11.077 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:11.077 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:11.077 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.077 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.077 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:11.077 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.077 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:11.077 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:11.077 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:11.078 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.078 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.051+0000 7f07cf7fe640 4 set_mon_vals no callback set 2024-05-15T12:43:11.078 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.052+0000 7f38367fc640 1 -- 172.21.15.117:0/1662615750 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f383801c070 con 0x7f384013dac0 2024-05-15T12:43:11.078 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.052+0000 7f38367fc640 1 --2- 172.21.15.117:0/1662615750 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f382406ceb0 0x7f382406f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.079 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.053+0000 7f3845a5a640 1 --2- 172.21.15.117:0/1662615750 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f382406ceb0 0x7f382406f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:11.079 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.053+0000 7f3845a5a640 10 monclient: get_auth_request con 0x7f382406ceb0 auth_method 0 2024-05-15T12:43:11.079 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.053+0000 7f38367fc640 10 client.4280.objecter ms_handle_connect 0x7f382406ceb0 2024-05-15T12:43:11.079 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.053+0000 7f3845a5a640 1 --2- 172.21.15.117:0/1662615750 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f382406ceb0 0x7f382406f370 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f383c036000 tx=0x7f383c030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:11.079 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.054+0000 7f6d33fff640 1 -- 172.21.15.117:0/3188308714 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f6d4806a1d0 con 0x7f6d400cfab0 2024-05-15T12:43:11.079 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.054+0000 7f6d33fff640 10 client.4293.objecter ms_dispatch 0x7f6d48000b90 osd_map(26..26 src has 1..26) v4 2024-05-15T12:43:11.079 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.054+0000 7f6d33fff640 3 client.4293.objecter handle_osd_map got epochs [26,26] > 0 2024-05-15T12:43:11.079 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.054+0000 7f6d33fff640 3 client.4293.objecter handle_osd_map decoding full epoch 26 2024-05-15T12:43:11.080 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.054+0000 7f6d33fff640 10 client.4293.objecter checking command 1 2024-05-15T12:43:11.080 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.054+0000 7f6d33fff640 1 --2- 172.21.15.117:0/3188308714 >> [v2:172.21.15.178:6800/1383036222,v1:172.21.15.178:6801/1383036222] conn(0x7f6d2c0732e0 0x7f6d2c075700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.080 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.054+0000 7f6d33fff640 20 client.4293.objecter _get_session s=0x7f6d2c072c10 osd=6 3 2024-05-15T12:43:11.080 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.054+0000 7f6d33fff640 20 client.4293.objecter put_session s=0x7f6d2c072c10 osd=6 3 2024-05-15T12:43:11.080 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.054+0000 7f6d33fff640 15 client.4293.objecter _session_command_op_remove -1 1 2024-05-15T12:43:11.080 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.054+0000 7f6d33fff640 20 client.4293.objecter _get_session s=0x7f6d2c072c10 osd=6 3 2024-05-15T12:43:11.080 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.054+0000 7f6d33fff640 20 client.4293.objecter get_session s=0x7f6d2c072c10 osd=6 3 2024-05-15T12:43:11.080 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.054+0000 7f6d33fff640 15 client.4293.objecter _session_command_op_assign 6 1 2024-05-15T12:43:11.080 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.054+0000 7f6d33fff640 20 client.4293.objecter put_session s=0x7f6d2c072c10 osd=6 4 2024-05-15T12:43:11.080 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.054+0000 7f6d33fff640 10 client.4293.objecter _send_command 1 2024-05-15T12:43:11.080 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.054+0000 7f6d33fff640 1 -- 172.21.15.117:0/3188308714 --> [v2:172.21.15.178:6800/1383036222,v1:172.21.15.178:6801/1383036222] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6d2c075db0 con 0x7f6d2c0732e0 2024-05-15T12:43:11.080 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.054+0000 7f6d33fff640 20 client.4293.objecter dump_active .. 0 homeless 2024-05-15T12:43:11.080 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.054+0000 7f6d33fff640 1 -- 172.21.15.117:0/3188308714 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6d480cfb40 con 0x7f6d400cfab0 2024-05-15T12:43:11.080 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.054+0000 7f6d33fff640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-05-15T12:43:11.081 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.055+0000 7f656bfff640 1 -- 172.21.15.117:0/3888178903 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f6574011de0 con 0x7f6578033590 2024-05-15T12:43:11.081 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.055+0000 7f656bfff640 10 client.4277.objecter ms_dispatch 0x7f6580000b90 osd_map(26..26 src has 1..26) v4 2024-05-15T12:43:11.081 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.055+0000 7f656bfff640 3 client.4277.objecter handle_osd_map got epochs [26,26] > 0 2024-05-15T12:43:11.081 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.055+0000 7f656bfff640 3 client.4277.objecter handle_osd_map decoding full epoch 26 2024-05-15T12:43:11.081 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.056+0000 7f656bfff640 10 client.4277.objecter checking command 1 2024-05-15T12:43:11.081 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.056+0000 7f656bfff640 1 --2- 172.21.15.117:0/3888178903 >> [v2:172.21.15.117:6818/3864975175,v1:172.21.15.117:6819/3864975175] conn(0x7f6564073320 0x7f6564075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.081 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.056+0000 7f656bfff640 20 client.4277.objecter _get_session s=0x7f6564072c50 osd=1 3 2024-05-15T12:43:11.081 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.056+0000 7f656bfff640 20 client.4277.objecter put_session s=0x7f6564072c50 osd=1 3 2024-05-15T12:43:11.081 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.056+0000 7f656bfff640 15 client.4277.objecter _session_command_op_remove -1 1 2024-05-15T12:43:11.081 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.056+0000 7f656bfff640 20 client.4277.objecter _get_session s=0x7f6564072c50 osd=1 3 2024-05-15T12:43:11.081 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.056+0000 7f656bfff640 20 client.4277.objecter get_session s=0x7f6564072c50 osd=1 3 2024-05-15T12:43:11.081 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.056+0000 7f656bfff640 15 client.4277.objecter _session_command_op_assign 1 1 2024-05-15T12:43:11.081 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.056+0000 7f656bfff640 20 client.4277.objecter put_session s=0x7f6564072c50 osd=1 4 2024-05-15T12:43:11.082 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.056+0000 7f656bfff640 10 client.4277.objecter _send_command 1 2024-05-15T12:43:11.082 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.056+0000 7f656bfff640 1 -- 172.21.15.117:0/3888178903 --> [v2:172.21.15.117:6818/3864975175,v1:172.21.15.117:6819/3864975175] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6564075df0 con 0x7f6564073320 2024-05-15T12:43:11.082 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.056+0000 7f656bfff640 20 client.4277.objecter dump_active .. 0 homeless 2024-05-15T12:43:11.082 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.055+0000 7f80157fa640 1 -- 172.21.15.117:0/3984485137 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8018098070 con 0x7f801002e200 2024-05-15T12:43:11.082 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.055+0000 7f80157fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:11.082 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.055+0000 7f80157fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T12:43:11.082 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.055+0000 7f80157fa640 10 monclient: dump: 2024-05-15T12:43:11.082 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:11.082 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:11.082 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.082 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.082 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:11.082 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.082 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:11.083 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:11.083 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:11.083 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.083 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.055+0000 7f80157fa640 10 monclient: _finish_auth 0 2024-05-15T12:43:11.083 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.055+0000 7f80157fa640 10 monclient: _check_auth_tickets 2024-05-15T12:43:11.083 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.055+0000 7f80157fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:11.083 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.055+0000 7f80157fa640 1 -- 172.21.15.117:0/3984485137 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8018067f90 con 0x7f801002e200 2024-05-15T12:43:11.083 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.055+0000 7f80157fa640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:11.083 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.056+0000 7f6d46ffd640 1 --2- 172.21.15.117:0/3188308714 >> [v2:172.21.15.178:6800/1383036222,v1:172.21.15.178:6801/1383036222] conn(0x7f6d2c0732e0 0x7f6d2c075700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:11.083 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.056+0000 7f6d46ffd640 10 monclient: get_auth_request con 0x7f6d2c0732e0 auth_method 0 2024-05-15T12:43:11.084 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.055+0000 7f80157fa640 1 -- 172.21.15.117:0/3984485137 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80180933f0 con 0x7f801002e200 2024-05-15T12:43:11.084 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.055+0000 7f80157fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:11.084 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.055+0000 7f80157fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T12:43:11.084 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.055+0000 7f80157fa640 10 monclient: dump: 2024-05-15T12:43:11.084 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:11.084 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:11.084 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.084 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.084 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:11.084 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.084 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:11.084 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:11.084 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:11.085 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.085 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.055+0000 7f801da08640 5 monclient: authenticate success, global_id 4296 2024-05-15T12:43:11.085 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.055+0000 7f801da08640 10 monclient: _renew_subs 2024-05-15T12:43:11.085 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.055+0000 7f8016ffd640 4 set_mon_vals no callback set 2024-05-15T12:43:11.085 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.055+0000 7f801da08640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:43:11.085 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.055+0000 7f801da08640 1 -- 172.21.15.117:0/3984485137 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80100c54e0 con 0x7f801002e200 2024-05-15T12:43:11.085 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.056+0000 7f801da08640 10 client.4296.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:43:11.085 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.056+0000 7f801da08640 10 monclient: _renew_subs 2024-05-15T12:43:11.085 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.056+0000 7f801da08640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:43:11.085 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.056+0000 7f801da08640 1 -- 172.21.15.117:0/3984485137 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80100ce950 con 0x7f801002e200 2024-05-15T12:43:11.085 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.056+0000 7f801da08640 1 librados: init done 2024-05-15T12:43:11.085 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.056+0000 7f657ffff640 1 --2- 172.21.15.117:0/3888178903 >> [v2:172.21.15.117:6818/3864975175,v1:172.21.15.117:6819/3864975175] conn(0x7f6564073320 0x7f6564075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:11.086 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.056+0000 7f801da08640 10 client.4296.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:11.086 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.057+0000 7f801da08640 15 client.4296.objecter _session_command_op_assign -1 1 2024-05-15T12:43:11.086 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.057+0000 7f801da08640 20 client.4296.objecter _get_session osd=-1 returning homeless 2024-05-15T12:43:11.086 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.057+0000 7f801da08640 10 client.4296.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:43:11.086 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.057+0000 7f801da08640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:43:11.086 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.057+0000 7f801da08640 1 -- 172.21.15.117:0/3984485137 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8010005390 con 0x7f801002e200 2024-05-15T12:43:11.086 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.057+0000 7f657ffff640 10 monclient: get_auth_request con 0x7f6564073320 auth_method 0 2024-05-15T12:43:11.087 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.060+0000 7f38367fc640 1 -- 172.21.15.117:0/1662615750 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f3838056fa0 con 0x7f384013dac0 2024-05-15T12:43:11.087 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.060+0000 7f38367fc640 10 client.4280.objecter ms_dispatch 0x7f3840000b90 osd_map(26..26 src has 1..26) v4 2024-05-15T12:43:11.087 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.060+0000 7f38367fc640 3 client.4280.objecter handle_osd_map got epochs [26,26] > 0 2024-05-15T12:43:11.087 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.060+0000 7f38367fc640 3 client.4280.objecter handle_osd_map decoding full epoch 26 2024-05-15T12:43:11.087 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.060+0000 7f38367fc640 10 client.4280.objecter checking command 1 2024-05-15T12:43:11.087 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.061+0000 7f38367fc640 1 --2- 172.21.15.117:0/1662615750 >> [v2:172.21.15.117:6804/1425835241,v1:172.21.15.117:6805/1425835241] conn(0x7f38240732f0 0x7f3824075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.087 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.061+0000 7f38367fc640 20 client.4280.objecter _get_session s=0x7f3824072c20 osd=2 3 2024-05-15T12:43:11.087 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.061+0000 7f6d33fff640 10 client.4293.objecter ms_handle_connect 0x7f6d2c0732e0 2024-05-15T12:43:11.087 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.061+0000 7f6d46ffd640 1 --2- 172.21.15.117:0/3188308714 >> [v2:172.21.15.178:6800/1383036222,v1:172.21.15.178:6801/1383036222] conn(0x7f6d2c0732e0 0x7f6d2c075700 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:11.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.061+0000 7f38367fc640 20 client.4280.objecter put_session s=0x7f3824072c20 osd=2 3 2024-05-15T12:43:11.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.061+0000 7f38367fc640 15 client.4280.objecter _session_command_op_remove -1 1 2024-05-15T12:43:11.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.061+0000 7f38367fc640 20 client.4280.objecter _get_session s=0x7f3824072c20 osd=2 3 2024-05-15T12:43:11.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.061+0000 7f38367fc640 20 client.4280.objecter get_session s=0x7f3824072c20 osd=2 3 2024-05-15T12:43:11.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.061+0000 7f38367fc640 15 client.4280.objecter _session_command_op_assign 2 1 2024-05-15T12:43:11.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.061+0000 7f38367fc640 20 client.4280.objecter put_session s=0x7f3824072c20 osd=2 4 2024-05-15T12:43:11.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.061+0000 7f38367fc640 10 client.4280.objecter _send_command 1 2024-05-15T12:43:11.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.061+0000 7f38367fc640 1 -- 172.21.15.117:0/1662615750 --> [v2:172.21.15.117:6804/1425835241,v1:172.21.15.117:6805/1425835241] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3838018ec0 con 0x7f38240732f0 2024-05-15T12:43:11.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.061+0000 7f38367fc640 20 client.4280.objecter dump_active .. 0 homeless 2024-05-15T12:43:11.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.061+0000 7f3845259640 1 --2- 172.21.15.117:0/1662615750 >> [v2:172.21.15.117:6804/1425835241,v1:172.21.15.117:6805/1425835241] conn(0x7f38240732f0 0x7f3824075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:11.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.062+0000 7f657ffff640 1 --2- 172.21.15.117:0/3888178903 >> [v2:172.21.15.117:6818/3864975175,v1:172.21.15.117:6819/3864975175] conn(0x7f6564073320 0x7f6564075740 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:11.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.062+0000 7f656bfff640 10 client.4277.objecter ms_handle_connect 0x7f6564073320 2024-05-15T12:43:11.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.062+0000 7f3845259640 10 monclient: get_auth_request con 0x7f38240732f0 auth_method 0 2024-05-15T12:43:11.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.066+0000 7f99417fa640 1 -- 172.21.15.117:0/363281595 <== osd.0 v2:172.21.15.117:6826/1673738625 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9938006780 con 0x7f99300732f0 2024-05-15T12:43:11.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.066+0000 7f99417fa640 10 client.4318.objecter ms_dispatch 0x7f9944000b90 command_reply(tid 1: 0 ) v1 2024-05-15T12:43:11.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.066+0000 7f99417fa640 10 client.4318.objecter _finish_command 1 = system:0 2024-05-15T12:43:11.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.066+0000 7f99417fa640 20 client.4318.objecter put_session s=0x7f9930072c20 osd=0 4 2024-05-15T12:43:11.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.066+0000 7f99417fa640 15 client.4318.objecter _session_command_op_remove 0 1 2024-05-15T12:43:11.091 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.067+0000 7f3845259640 1 --2- 172.21.15.117:0/1662615750 >> [v2:172.21.15.117:6804/1425835241,v1:172.21.15.117:6805/1425835241] conn(0x7f38240732f0 0x7f3824075710 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:11.091 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.072+0000 7f38367fc640 10 client.4280.objecter ms_handle_connect 0x7f38240732f0 2024-05-15T12:43:11.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.077+0000 7f07cdffb640 1 -- 172.21.15.117:0/2001633136 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f07d0046070 con 0x7f07d81db8e0 2024-05-15T12:43:11.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.077+0000 7f07cdffb640 1 --2- 172.21.15.117:0/2001633136 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f07bc06ce60 0x7f07bc06f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.078+0000 7f07dd07f640 1 --2- 172.21.15.117:0/2001633136 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f07bc06ce60 0x7f07bc06f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:11.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.078+0000 7f07dd07f640 10 monclient: get_auth_request con 0x7f07bc06ce60 auth_method 0 2024-05-15T12:43:11.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.078+0000 7f07cdffb640 10 client.4342.objecter ms_handle_connect 0x7f07bc06ce60 2024-05-15T12:43:11.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.078+0000 7f07dd07f640 1 --2- 172.21.15.117:0/2001633136 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f07bc06ce60 0x7f07bc06f320 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f07d4007fd0 tx=0x7f07d4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:11.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.080+0000 7f07cdffb640 1 -- 172.21.15.117:0/2001633136 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f07d00787b0 con 0x7f07d81db8e0 2024-05-15T12:43:11.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.080+0000 7f07cdffb640 10 client.4342.objecter ms_dispatch 0x7f07d8000b90 osd_map(26..26 src has 1..26) v4 2024-05-15T12:43:11.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.080+0000 7f07cdffb640 3 client.4342.objecter handle_osd_map got epochs [26,26] > 0 2024-05-15T12:43:11.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.080+0000 7f07cdffb640 3 client.4342.objecter handle_osd_map decoding full epoch 26 2024-05-15T12:43:11.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.080+0000 7f07cdffb640 10 client.4342.objecter checking command 1 2024-05-15T12:43:11.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.080+0000 7f07cdffb640 1 --2- 172.21.15.117:0/2001633136 >> [v2:172.21.15.178:6808/3049776526,v1:172.21.15.178:6809/3049776526] conn(0x7f07bc0732a0 0x7f07bc0756c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.080+0000 7f07cdffb640 20 client.4342.objecter _get_session s=0x7f07bc072bd0 osd=7 3 2024-05-15T12:43:11.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.080+0000 7f07cdffb640 20 client.4342.objecter put_session s=0x7f07bc072bd0 osd=7 3 2024-05-15T12:43:11.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.080+0000 7f07cdffb640 15 client.4342.objecter _session_command_op_remove -1 1 2024-05-15T12:43:11.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.080+0000 7f07cdffb640 20 client.4342.objecter _get_session s=0x7f07bc072bd0 osd=7 3 2024-05-15T12:43:11.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.080+0000 7f07cdffb640 20 client.4342.objecter get_session s=0x7f07bc072bd0 osd=7 3 2024-05-15T12:43:11.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.080+0000 7f07cdffb640 15 client.4342.objecter _session_command_op_assign 7 1 2024-05-15T12:43:11.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.080+0000 7f07cdffb640 20 client.4342.objecter put_session s=0x7f07bc072bd0 osd=7 4 2024-05-15T12:43:11.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.080+0000 7f07cdffb640 10 client.4342.objecter _send_command 1 2024-05-15T12:43:11.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.080+0000 7f07cdffb640 1 -- 172.21.15.117:0/2001633136 --> [v2:172.21.15.178:6808/3049776526,v1:172.21.15.178:6809/3049776526] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f07bc075d70 con 0x7f07bc0732a0 2024-05-15T12:43:11.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.080+0000 7f07cdffb640 20 client.4342.objecter dump_active .. 0 homeless 2024-05-15T12:43:11.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.081+0000 7f07dc87e640 1 --2- 172.21.15.117:0/2001633136 >> [v2:172.21.15.178:6808/3049776526,v1:172.21.15.178:6809/3049776526] conn(0x7f07bc0732a0 0x7f07bc0756c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:11.094 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.081+0000 7f07dc87e640 10 monclient: get_auth_request con 0x7f07bc0732a0 auth_method 0 2024-05-15T12:43:11.094 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.086+0000 7f07cdffb640 10 client.4342.objecter ms_handle_connect 0x7f07bc0732a0 2024-05-15T12:43:11.094 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.086+0000 7f07dc87e640 1 --2- 172.21.15.117:0/2001633136 >> [v2:172.21.15.178:6808/3049776526,v1:172.21.15.178:6809/3049776526] conn(0x7f07bc0732a0 0x7f07bc0756c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:11.094 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.087+0000 7f994aed6640 10 client.4318.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-05-15T12:43:11.094 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.087+0000 7f994aed6640 15 client.4318.objecter _session_command_op_assign -1 2 2024-05-15T12:43:11.094 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.087+0000 7f994aed6640 20 client.4318.objecter _get_session s=0x7f9930072c20 osd=0 3 2024-05-15T12:43:11.094 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.087+0000 7f994aed6640 20 client.4318.objecter put_session s=0x7f9930072c20 osd=0 3 2024-05-15T12:43:11.095 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.087+0000 7f994aed6640 20 client.4318.objecter _get_session s=0x7f9930072c20 osd=0 3 2024-05-15T12:43:11.095 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.087+0000 7f994aed6640 15 client.4318.objecter _session_command_op_remove -1 2 2024-05-15T12:43:11.095 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.087+0000 7f994aed6640 20 client.4318.objecter get_session s=0x7f9930072c20 osd=0 3 2024-05-15T12:43:11.095 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.087+0000 7f994aed6640 15 client.4318.objecter _session_command_op_assign 0 2 2024-05-15T12:43:11.095 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.087+0000 7f994aed6640 20 client.4318.objecter put_session s=0x7f9930072c20 osd=0 4 2024-05-15T12:43:11.095 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.087+0000 7f994aed6640 10 client.4318.objecter _send_command 2 2024-05-15T12:43:11.095 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.087+0000 7f994aed6640 1 -- 172.21.15.117:0/363281595 --> [v2:172.21.15.117:6826/1673738625,v1:172.21.15.117:6827/1673738625] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f994404f0e0 con 0x7f99300732f0 2024-05-15T12:43:11.097 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.107+0000 7f99417fa640 1 -- 172.21.15.117:0/363281595 <== osd.0 v2:172.21.15.117:6826/1673738625 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f994404f0e0 con 0x7f99300732f0 2024-05-15T12:43:11.097 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.107+0000 7f99417fa640 10 client.4318.objecter ms_dispatch 0x7f9944000b90 command_reply(tid 2: 0 ) v1 2024-05-15T12:43:11.097 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.107+0000 7f99417fa640 10 client.4318.objecter _finish_command 2 = system:0 2024-05-15T12:43:11.097 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.107+0000 7f99417fa640 20 client.4318.objecter put_session s=0x7f9930072c20 osd=0 4 2024-05-15T12:43:11.097 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.107+0000 7f99417fa640 15 client.4318.objecter _session_command_op_remove 0 2 2024-05-15T12:43:11.098 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.107+0000 7f994aed6640 10 librados: watch_flush enter 2024-05-15T12:43:11.098 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.108+0000 7f994aed6640 10 librados: watch_flush exit 2024-05-15T12:43:11.099 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.108+0000 7f994aed6640 10 client.4318.objecter close_session for osd.0 2024-05-15T12:43:11.099 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.108+0000 7f994aed6640 1 -- 172.21.15.117:0/363281595 >> [v2:172.21.15.117:6826/1673738625,v1:172.21.15.117:6827/1673738625] conn(0x7f99300732f0 msgr2=0x7f9930075710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.099 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.108+0000 7f994aed6640 1 --2- 172.21.15.117:0/363281595 >> [v2:172.21.15.117:6826/1673738625,v1:172.21.15.117:6827/1673738625] conn(0x7f99300732f0 0x7f9930075710 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.099 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.108+0000 7f994aed6640 20 client.4318.objecter put_session s=0x7f9930072c20 osd=0 1 2024-05-15T12:43:11.099 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.108+0000 7f994aed6640 20 client.4318.objecter shutdown clearing up homeless session... 2024-05-15T12:43:11.099 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.108+0000 7f994aed6640 10 client.4318.objecter successfully canceled tick 2024-05-15T12:43:11.099 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.108+0000 7f994aed6640 1 -- 172.21.15.117:0/363281595 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f993006ceb0 msgr2=0x7f993006f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.099 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.108+0000 7f994aed6640 1 --2- 172.21.15.117:0/363281595 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f993006ceb0 0x7f993006f370 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f993c035000 tx=0x7f993c02e000 comp rx=0 tx=0).stop 2024-05-15T12:43:11.099 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.108+0000 7f994aed6640 10 monclient: shutdown 2024-05-15T12:43:11.099 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.108+0000 7f994aed6640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:11.099 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.108+0000 7f994aed6640 1 -- 172.21.15.117:0/363281595 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f99440716a0 msgr2=0x7f994414d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.099 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.108+0000 7f994aed6640 1 --2- 172.21.15.117:0/363281595 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f99440716a0 0x7f994414d790 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f993800b410 tx=0x7f993800b8d0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.100 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.108+0000 7f9949473640 1 -- 172.21.15.117:0/363281595 reap_dead start 2024-05-15T12:43:11.100 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.108+0000 7f994aed6640 1 -- 172.21.15.117:0/363281595 shutdown_connections 2024-05-15T12:43:11.100 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.108+0000 7f994aed6640 1 -- 172.21.15.117:0/363281595 >> 172.21.15.117:0/363281595 conn(0x7f994406ada0 msgr2=0x7f9944071ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:11.100 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.108+0000 7f994aed6640 1 -- 172.21.15.117:0/363281595 shutdown_connections 2024-05-15T12:43:11.100 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.108+0000 7f994aed6640 1 -- 172.21.15.117:0/363281595 wait complete. 2024-05-15T12:43:11.100 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.108+0000 7f994aed6640 1 librados: shutdown 2024-05-15T12:43:11.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.117+0000 7f9e15ffb640 1 -- 172.21.15.117:0/2577752866 <== osd.4 v2:172.21.15.178:6824/248472851 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9e18010e50 con 0x7f9e040732f0 2024-05-15T12:43:11.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.117+0000 7f9e15ffb640 10 client.4339.objecter ms_dispatch 0x7f9e20000b90 command_reply(tid 1: 0 ) v1 2024-05-15T12:43:11.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.117+0000 7f9e15ffb640 10 client.4339.objecter _finish_command 1 = system:0 2024-05-15T12:43:11.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.117+0000 7f9e15ffb640 20 client.4339.objecter put_session s=0x7f9e04072c20 osd=4 4 2024-05-15T12:43:11.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.117+0000 7f9e15ffb640 15 client.4339.objecter _session_command_op_remove 4 1 2024-05-15T12:43:11.113 INFO:teuthology.orchestra.run.smithi117.stdout:85899345931 2024-05-15T12:43:11.113 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-05-15T12:43:11.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.136+0000 7f6d33fff640 1 -- 172.21.15.117:0/3188308714 <== osd.6 v2:172.21.15.178:6800/1383036222 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6d2c075db0 con 0x7f6d2c0732e0 2024-05-15T12:43:11.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.136+0000 7f6d33fff640 10 client.4293.objecter ms_dispatch 0x7f6d48000b90 command_reply(tid 1: 0 ) v1 2024-05-15T12:43:11.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.136+0000 7f6d33fff640 10 client.4293.objecter _finish_command 1 = system:0 2024-05-15T12:43:11.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.136+0000 7f6d33fff640 20 client.4293.objecter put_session s=0x7f6d2c072c10 osd=6 4 2024-05-15T12:43:11.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.136+0000 7f6d33fff640 15 client.4293.objecter _session_command_op_remove 6 1 2024-05-15T12:43:11.129 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.138+0000 7f07cdffb640 1 -- 172.21.15.117:0/2001633136 <== osd.7 v2:172.21.15.178:6808/3049776526 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f07bc075d70 con 0x7f07bc0732a0 2024-05-15T12:43:11.129 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.138+0000 7f07cdffb640 10 client.4342.objecter ms_dispatch 0x7f07d8000b90 command_reply(tid 1: 0 ) v1 2024-05-15T12:43:11.129 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.138+0000 7f07cdffb640 10 client.4342.objecter _finish_command 1 = system:0 2024-05-15T12:43:11.129 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.138+0000 7f07cdffb640 20 client.4342.objecter put_session s=0x7f07bc072bd0 osd=7 4 2024-05-15T12:43:11.129 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.138+0000 7f07cdffb640 15 client.4342.objecter _session_command_op_remove 7 1 2024-05-15T12:43:11.139 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.149+0000 7f9e267d8640 10 client.4339.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-05-15T12:43:11.139 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.149+0000 7f9e267d8640 15 client.4339.objecter _session_command_op_assign -1 2 2024-05-15T12:43:11.139 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.149+0000 7f9e267d8640 20 client.4339.objecter _get_session s=0x7f9e04072c20 osd=4 3 2024-05-15T12:43:11.139 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.149+0000 7f9e267d8640 20 client.4339.objecter put_session s=0x7f9e04072c20 osd=4 3 2024-05-15T12:43:11.139 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.149+0000 7f9e267d8640 20 client.4339.objecter _get_session s=0x7f9e04072c20 osd=4 3 2024-05-15T12:43:11.139 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.149+0000 7f9e267d8640 15 client.4339.objecter _session_command_op_remove -1 2 2024-05-15T12:43:11.140 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.149+0000 7f9e267d8640 20 client.4339.objecter get_session s=0x7f9e04072c20 osd=4 3 2024-05-15T12:43:11.140 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.149+0000 7f9e267d8640 15 client.4339.objecter _session_command_op_assign 4 2 2024-05-15T12:43:11.140 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.149+0000 7f9e267d8640 20 client.4339.objecter put_session s=0x7f9e04072c20 osd=4 4 2024-05-15T12:43:11.140 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.149+0000 7f9e267d8640 10 client.4339.objecter _send_command 2 2024-05-15T12:43:11.140 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.149+0000 7f9e267d8640 1 -- 172.21.15.117:0/2577752866 --> [v2:172.21.15.178:6824/248472851,v1:172.21.15.178:6825/248472851] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9e20066ec0 con 0x7f9e040732f0 2024-05-15T12:43:11.142 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.152+0000 7f38367fc640 1 -- 172.21.15.117:0/1662615750 <== osd.2 v2:172.21.15.117:6804/1425835241 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3838018ec0 con 0x7f38240732f0 2024-05-15T12:43:11.142 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.152+0000 7f38367fc640 10 client.4280.objecter ms_dispatch 0x7f3840000b90 command_reply(tid 1: 0 ) v1 2024-05-15T12:43:11.142 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.152+0000 7f38367fc640 10 client.4280.objecter _finish_command 1 = system:0 2024-05-15T12:43:11.143 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.152+0000 7f38367fc640 20 client.4280.objecter put_session s=0x7f3824072c20 osd=2 4 2024-05-15T12:43:11.143 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.152+0000 7f38367fc640 15 client.4280.objecter _session_command_op_remove 2 1 2024-05-15T12:43:11.149 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.159+0000 7f9e15ffb640 1 -- 172.21.15.117:0/2577752866 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9e18055060 con 0x7f9e20149ca0 2024-05-15T12:43:11.149 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.159+0000 7f9e15ffb640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-05-15T12:43:11.150 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.160+0000 7f07cdffb640 1 -- 172.21.15.117:0/2001633136 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f07d00769b0 con 0x7f07d81db8e0 2024-05-15T12:43:11.150 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.160+0000 7f07cdffb640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-05-15T12:43:11.150 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.160+0000 7f9e15ffb640 1 -- 172.21.15.117:0/2577752866 <== osd.4 v2:172.21.15.178:6824/248472851 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f9e20066ec0 con 0x7f9e040732f0 2024-05-15T12:43:11.150 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.160+0000 7f9e15ffb640 10 client.4339.objecter ms_dispatch 0x7f9e20000b90 command_reply(tid 2: 0 ) v1 2024-05-15T12:43:11.150 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.160+0000 7f9e15ffb640 10 client.4339.objecter _finish_command 2 = system:0 2024-05-15T12:43:11.150 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.160+0000 7f9e15ffb640 20 client.4339.objecter put_session s=0x7f9e04072c20 osd=4 4 2024-05-15T12:43:11.150 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.160+0000 7f9e15ffb640 15 client.4339.objecter _session_command_op_remove 4 2 2024-05-15T12:43:11.151 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.161+0000 7f9e267d8640 10 librados: watch_flush enter 2024-05-15T12:43:11.151 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.161+0000 7f9e267d8640 10 librados: watch_flush exit 2024-05-15T12:43:11.151 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.161+0000 7f9e267d8640 10 client.4339.objecter close_session for osd.4 2024-05-15T12:43:11.151 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.161+0000 7f9e267d8640 1 -- 172.21.15.117:0/2577752866 >> [v2:172.21.15.178:6824/248472851,v1:172.21.15.178:6825/248472851] conn(0x7f9e040732f0 msgr2=0x7f9e04075710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.151 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.161+0000 7f9e267d8640 1 --2- 172.21.15.117:0/2577752866 >> [v2:172.21.15.178:6824/248472851,v1:172.21.15.178:6825/248472851] conn(0x7f9e040732f0 0x7f9e04075710 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.151 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.161+0000 7f9e267d8640 20 client.4339.objecter put_session s=0x7f9e04072c20 osd=4 1 2024-05-15T12:43:11.151 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.161+0000 7f9e267d8640 20 client.4339.objecter shutdown clearing up homeless session... 2024-05-15T12:43:11.151 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.161+0000 7f9e267d8640 10 client.4339.objecter successfully canceled tick 2024-05-15T12:43:11.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.161+0000 7f9e267d8640 1 -- 172.21.15.117:0/2577752866 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f9e0406ceb0 msgr2=0x7f9e0406f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.161+0000 7f9e267d8640 1 --2- 172.21.15.117:0/2577752866 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f9e0406ceb0 0x7f9e0406f370 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c009650 tx=0x7f9e1c006ba0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.161+0000 7f9e267d8640 10 monclient: shutdown 2024-05-15T12:43:11.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.161+0000 7f9e267d8640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:11.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.161+0000 7f9e267d8640 1 -- 172.21.15.117:0/2577752866 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9e20149ca0 msgr2=0x7f9e20148330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.161+0000 7f9e267d8640 1 --2- 172.21.15.117:0/2577752866 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9e20149ca0 0x7f9e20148330 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9e1800b9a0 tx=0x7f9e1800be60 comp rx=0 tx=0).stop 2024-05-15T12:43:11.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.161+0000 7f9e25fd7640 1 -- 172.21.15.117:0/2577752866 reap_dead start 2024-05-15T12:43:11.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.161+0000 7f9e267d8640 1 -- 172.21.15.117:0/2577752866 shutdown_connections 2024-05-15T12:43:11.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.161+0000 7f9e267d8640 1 -- 172.21.15.117:0/2577752866 >> 172.21.15.117:0/2577752866 conn(0x7f9e200b1640 msgr2=0x7f9e200b04f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:11.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.163+0000 7f9e267d8640 1 -- 172.21.15.117:0/2577752866 shutdown_connections 2024-05-15T12:43:11.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.163+0000 7f9e267d8640 1 -- 172.21.15.117:0/2577752866 wait complete. 2024-05-15T12:43:11.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.163+0000 7f9e267d8640 1 librados: shutdown 2024-05-15T12:43:11.155 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.165+0000 7f656bfff640 1 -- 172.21.15.117:0/3888178903 <== osd.1 v2:172.21.15.117:6818/3864975175 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6564075df0 con 0x7f6564073320 2024-05-15T12:43:11.155 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.165+0000 7f656bfff640 10 client.4277.objecter ms_dispatch 0x7f6580000b90 command_reply(tid 1: 0 ) v1 2024-05-15T12:43:11.155 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.165+0000 7f656bfff640 10 client.4277.objecter _finish_command 1 = system:0 2024-05-15T12:43:11.155 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.165+0000 7f656bfff640 20 client.4277.objecter put_session s=0x7f6564072c50 osd=1 4 2024-05-15T12:43:11.155 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.165+0000 7f656bfff640 15 client.4277.objecter _session_command_op_remove 1 1 2024-05-15T12:43:11.156 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.166+0000 7f6d4c9d3640 10 client.4293.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-05-15T12:43:11.156 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.166+0000 7f6d4c9d3640 15 client.4293.objecter _session_command_op_assign -1 2 2024-05-15T12:43:11.156 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.166+0000 7f6d4c9d3640 20 client.4293.objecter _get_session s=0x7f6d2c072c10 osd=6 3 2024-05-15T12:43:11.156 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.166+0000 7f6d4c9d3640 20 client.4293.objecter put_session s=0x7f6d2c072c10 osd=6 3 2024-05-15T12:43:11.156 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.166+0000 7f6d4c9d3640 20 client.4293.objecter _get_session s=0x7f6d2c072c10 osd=6 3 2024-05-15T12:43:11.157 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.166+0000 7f6d4c9d3640 15 client.4293.objecter _session_command_op_remove -1 2 2024-05-15T12:43:11.157 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.166+0000 7f6d4c9d3640 20 client.4293.objecter get_session s=0x7f6d2c072c10 osd=6 3 2024-05-15T12:43:11.157 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.166+0000 7f6d4c9d3640 15 client.4293.objecter _session_command_op_assign 6 2 2024-05-15T12:43:11.157 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.166+0000 7f6d4c9d3640 20 client.4293.objecter put_session s=0x7f6d2c072c10 osd=6 4 2024-05-15T12:43:11.157 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.166+0000 7f6d4c9d3640 10 client.4293.objecter _send_command 2 2024-05-15T12:43:11.157 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.166+0000 7f6d4c9d3640 1 -- 172.21.15.117:0/3188308714 --> [v2:172.21.15.178:6800/1383036222,v1:172.21.15.178:6801/1383036222] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6d400057e0 con 0x7f6d2c0732e0 2024-05-15T12:43:11.158 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.168+0000 7f07df2e3640 10 client.4342.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-05-15T12:43:11.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.168+0000 7f07df2e3640 15 client.4342.objecter _session_command_op_assign -1 2 2024-05-15T12:43:11.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.168+0000 7f07df2e3640 20 client.4342.objecter _get_session s=0x7f07bc072bd0 osd=7 3 2024-05-15T12:43:11.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.168+0000 7f07df2e3640 20 client.4342.objecter put_session s=0x7f07bc072bd0 osd=7 3 2024-05-15T12:43:11.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.168+0000 7f07df2e3640 20 client.4342.objecter _get_session s=0x7f07bc072bd0 osd=7 3 2024-05-15T12:43:11.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.168+0000 7f07df2e3640 15 client.4342.objecter _session_command_op_remove -1 2 2024-05-15T12:43:11.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.168+0000 7f07df2e3640 20 client.4342.objecter get_session s=0x7f07bc072bd0 osd=7 3 2024-05-15T12:43:11.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.168+0000 7f07df2e3640 15 client.4342.objecter _session_command_op_assign 7 2 2024-05-15T12:43:11.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.168+0000 7f07df2e3640 20 client.4342.objecter put_session s=0x7f07bc072bd0 osd=7 4 2024-05-15T12:43:11.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.168+0000 7f07df2e3640 10 client.4342.objecter _send_command 2 2024-05-15T12:43:11.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.168+0000 7f07df2e3640 1 -- 172.21.15.117:0/2001633136 --> [v2:172.21.15.178:6808/3049776526,v1:172.21.15.178:6809/3049776526] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f07d80acb40 con 0x7f07bc0732a0 2024-05-15T12:43:11.165 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.175+0000 7f3847cbe640 10 client.4280.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-05-15T12:43:11.166 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.175+0000 7f3847cbe640 15 client.4280.objecter _session_command_op_assign -1 2 2024-05-15T12:43:11.166 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.175+0000 7f3847cbe640 20 client.4280.objecter _get_session s=0x7f3824072c20 osd=2 3 2024-05-15T12:43:11.166 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.175+0000 7f3847cbe640 20 client.4280.objecter put_session s=0x7f3824072c20 osd=2 3 2024-05-15T12:43:11.166 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.175+0000 7f3847cbe640 20 client.4280.objecter _get_session s=0x7f3824072c20 osd=2 3 2024-05-15T12:43:11.166 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.175+0000 7f3847cbe640 15 client.4280.objecter _session_command_op_remove -1 2 2024-05-15T12:43:11.166 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.175+0000 7f3847cbe640 20 client.4280.objecter get_session s=0x7f3824072c20 osd=2 3 2024-05-15T12:43:11.166 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.175+0000 7f3847cbe640 15 client.4280.objecter _session_command_op_assign 2 2 2024-05-15T12:43:11.166 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.175+0000 7f3847cbe640 20 client.4280.objecter put_session s=0x7f3824072c20 osd=2 4 2024-05-15T12:43:11.166 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.175+0000 7f3847cbe640 10 client.4280.objecter _send_command 2 2024-05-15T12:43:11.166 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.175+0000 7f3847cbe640 1 -- 172.21.15.117:0/1662615750 --> [v2:172.21.15.117:6804/1425835241,v1:172.21.15.117:6805/1425835241] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f384013efb0 con 0x7f38240732f0 2024-05-15T12:43:11.167 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.176+0000 7f07cdffb640 1 -- 172.21.15.117:0/2001633136 <== osd.7 v2:172.21.15.178:6808/3049776526 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f07c8004040 con 0x7f07bc0732a0 2024-05-15T12:43:11.167 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.176+0000 7f07cdffb640 10 client.4342.objecter ms_dispatch 0x7f07d8000b90 command_reply(tid 2: 0 ) v1 2024-05-15T12:43:11.167 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.176+0000 7f07cdffb640 10 client.4342.objecter _finish_command 2 = system:0 2024-05-15T12:43:11.167 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.176+0000 7f07cdffb640 20 client.4342.objecter put_session s=0x7f07bc072bd0 osd=7 4 2024-05-15T12:43:11.167 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.176+0000 7f07cdffb640 15 client.4342.objecter _session_command_op_remove 7 2 2024-05-15T12:43:11.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.176+0000 7f07df2e3640 10 librados: watch_flush enter 2024-05-15T12:43:11.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.177+0000 7f07df2e3640 10 librados: watch_flush exit 2024-05-15T12:43:11.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.177+0000 7f07df2e3640 10 client.4342.objecter close_session for osd.7 2024-05-15T12:43:11.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.177+0000 7f07df2e3640 1 -- 172.21.15.117:0/2001633136 >> [v2:172.21.15.178:6808/3049776526,v1:172.21.15.178:6809/3049776526] conn(0x7f07bc0732a0 msgr2=0x7f07bc0756c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.177+0000 7f07df2e3640 1 --2- 172.21.15.117:0/2001633136 >> [v2:172.21.15.178:6808/3049776526,v1:172.21.15.178:6809/3049776526] conn(0x7f07bc0732a0 0x7f07bc0756c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.177+0000 7f07df2e3640 20 client.4342.objecter put_session s=0x7f07bc072bd0 osd=7 1 2024-05-15T12:43:11.171 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.177+0000 7f07df2e3640 20 client.4342.objecter shutdown clearing up homeless session... 2024-05-15T12:43:11.171 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.177+0000 7f07df2e3640 10 client.4342.objecter successfully canceled tick 2024-05-15T12:43:11.171 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.177+0000 7f07df2e3640 1 -- 172.21.15.117:0/2001633136 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f07bc06ce60 msgr2=0x7f07bc06f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.171 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.177+0000 7f07df2e3640 1 --2- 172.21.15.117:0/2001633136 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f07bc06ce60 0x7f07bc06f320 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f07d4007fd0 tx=0x7f07d4009040 comp rx=0 tx=0).stop 2024-05-15T12:43:11.171 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.177+0000 7f07df2e3640 10 monclient: shutdown 2024-05-15T12:43:11.171 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.177+0000 7f07df2e3640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:11.171 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.177+0000 7f07df2e3640 1 -- 172.21.15.117:0/2001633136 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f07d81db8e0 msgr2=0x7f07d81d7830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.171 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.177+0000 7f07df2e3640 1 --2- 172.21.15.117:0/2001633136 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f07d81db8e0 0x7f07d81d7830 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f07d002cbc0 tx=0x7f07d80ab380 comp rx=0 tx=0).stop 2024-05-15T12:43:11.171 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.177+0000 7f07dd880640 1 -- 172.21.15.117:0/2001633136 reap_dead start 2024-05-15T12:43:11.171 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.177+0000 7f07df2e3640 1 -- 172.21.15.117:0/2001633136 shutdown_connections 2024-05-15T12:43:11.171 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.177+0000 7f07df2e3640 1 -- 172.21.15.117:0/2001633136 >> 172.21.15.117:0/2001633136 conn(0x7f07d8066cc0 msgr2=0x7f07d813f2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:11.171 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.177+0000 7f07df2e3640 1 -- 172.21.15.117:0/2001633136 shutdown_connections 2024-05-15T12:43:11.171 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.177+0000 7f07df2e3640 1 -- 172.21.15.117:0/2001633136 wait complete. 2024-05-15T12:43:11.171 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.177+0000 7f07df2e3640 1 librados: shutdown 2024-05-15T12:43:11.172 INFO:teuthology.orchestra.run.smithi117.stdout:81604378634 2024-05-15T12:43:11.172 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-05-15T12:43:11.173 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.179+0000 7f6d33fff640 1 -- 172.21.15.117:0/3188308714 <== osd.6 v2:172.21.15.178:6800/1383036222 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6d400057e0 con 0x7f6d2c0732e0 2024-05-15T12:43:11.173 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.179+0000 7f6d33fff640 10 client.4293.objecter ms_dispatch 0x7f6d48000b90 command_reply(tid 2: 0 ) v1 2024-05-15T12:43:11.173 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.179+0000 7f6d33fff640 10 client.4293.objecter _finish_command 2 = system:0 2024-05-15T12:43:11.173 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.179+0000 7f6d33fff640 20 client.4293.objecter put_session s=0x7f6d2c072c10 osd=6 4 2024-05-15T12:43:11.173 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.179+0000 7f6d33fff640 15 client.4293.objecter _session_command_op_remove 6 2 2024-05-15T12:43:11.174 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.179+0000 7f6d4c9d3640 10 librados: watch_flush enter 2024-05-15T12:43:11.175 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.179+0000 7f6d4c9d3640 10 librados: watch_flush exit 2024-05-15T12:43:11.175 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.179+0000 7f6d4c9d3640 10 client.4293.objecter close_session for osd.6 2024-05-15T12:43:11.175 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.179+0000 7f6d4c9d3640 1 -- 172.21.15.117:0/3188308714 >> [v2:172.21.15.178:6800/1383036222,v1:172.21.15.178:6801/1383036222] conn(0x7f6d2c0732e0 msgr2=0x7f6d2c075700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.175 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.179+0000 7f6d4c9d3640 1 --2- 172.21.15.117:0/3188308714 >> [v2:172.21.15.178:6800/1383036222,v1:172.21.15.178:6801/1383036222] conn(0x7f6d2c0732e0 0x7f6d2c075700 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.175 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.179+0000 7f6d4c9d3640 20 client.4293.objecter put_session s=0x7f6d2c072c10 osd=6 1 2024-05-15T12:43:11.175 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.179+0000 7f6d4c9d3640 20 client.4293.objecter shutdown clearing up homeless session... 2024-05-15T12:43:11.175 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.179+0000 7f6d4c9d3640 10 client.4293.objecter successfully canceled tick 2024-05-15T12:43:11.175 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.180+0000 7f6d4c9d3640 1 -- 172.21.15.117:0/3188308714 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f6d2c06cea0 msgr2=0x7f6d2c06f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.175 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.180+0000 7f6d4c9d3640 1 --2- 172.21.15.117:0/3188308714 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f6d2c06cea0 0x7f6d2c06f360 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c036000 tx=0x7f6d3c030000 comp rx=0 tx=0).stop 2024-05-15T12:43:11.175 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.180+0000 7f6d4c9d3640 10 monclient: shutdown 2024-05-15T12:43:11.175 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.180+0000 7f6d4c9d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:11.175 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.180+0000 7f6d4c9d3640 1 -- 172.21.15.117:0/3188308714 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6d400cfab0 msgr2=0x7f6d401652a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.175 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.180+0000 7f6d4c9d3640 1 --2- 172.21.15.117:0/3188308714 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6d400cfab0 0x7f6d401652a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6d4806a9b0 tx=0x7f6d4806a0a0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.176 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.180+0000 7f6d47fff640 1 -- 172.21.15.117:0/3188308714 reap_dead start 2024-05-15T12:43:11.176 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.180+0000 7f6d4c9d3640 1 -- 172.21.15.117:0/3188308714 shutdown_connections 2024-05-15T12:43:11.176 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.180+0000 7f6d4c9d3640 1 -- 172.21.15.117:0/3188308714 >> 172.21.15.117:0/3188308714 conn(0x7f6d400061a0 msgr2=0x7f6d400cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:11.176 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.180+0000 7f6d4c9d3640 1 -- 172.21.15.117:0/3188308714 shutdown_connections 2024-05-15T12:43:11.176 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.180+0000 7f6d4c9d3640 1 -- 172.21.15.117:0/3188308714 wait complete. 2024-05-15T12:43:11.176 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.180+0000 7f6d4c9d3640 1 librados: shutdown 2024-05-15T12:43:11.177 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.181+0000 7f80157fa640 1 -- 172.21.15.117:0/3984485137 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f801809b070 con 0x7f801002e200 2024-05-15T12:43:11.177 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.181+0000 7f80157fa640 1 --2- 172.21.15.117:0/3984485137 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f7ffc06ceb0 0x7f7ffc06f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.177 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.182+0000 7f8017fff640 1 --2- 172.21.15.117:0/3984485137 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f7ffc06ceb0 0x7f7ffc06f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:11.177 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.182+0000 7f8017fff640 10 monclient: get_auth_request con 0x7f7ffc06ceb0 auth_method 0 2024-05-15T12:43:11.177 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.182+0000 7f80157fa640 10 client.4296.objecter ms_handle_connect 0x7f7ffc06ceb0 2024-05-15T12:43:11.177 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.182+0000 7f8017fff640 1 --2- 172.21.15.117:0/3984485137 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f7ffc06ceb0 0x7f7ffc06f370 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8004036000 tx=0x7f8004030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:11.179 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.188+0000 7f80157fa640 1 -- 172.21.15.117:0/3984485137 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f80180d11e0 con 0x7f801002e200 2024-05-15T12:43:11.179 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.188+0000 7f80157fa640 10 client.4296.objecter ms_dispatch 0x7f8018000b90 osd_map(26..26 src has 1..26) v4 2024-05-15T12:43:11.179 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.188+0000 7f80157fa640 3 client.4296.objecter handle_osd_map got epochs [26,26] > 0 2024-05-15T12:43:11.179 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.188+0000 7f80157fa640 3 client.4296.objecter handle_osd_map decoding full epoch 26 2024-05-15T12:43:11.179 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.188+0000 7f80157fa640 10 client.4296.objecter checking command 1 2024-05-15T12:43:11.179 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.188+0000 7f80157fa640 1 --2- 172.21.15.117:0/3984485137 >> [v2:172.21.15.117:6802/3842021926,v1:172.21.15.117:6803/3842021926] conn(0x7f7ffc0732f0 0x7f7ffc075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.179 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.188+0000 7f80157fa640 20 client.4296.objecter _get_session s=0x7f7ffc072c20 osd=3 3 2024-05-15T12:43:11.179 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.188+0000 7f80157fa640 20 client.4296.objecter put_session s=0x7f7ffc072c20 osd=3 3 2024-05-15T12:43:11.179 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.188+0000 7f80157fa640 15 client.4296.objecter _session_command_op_remove -1 1 2024-05-15T12:43:11.179 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.188+0000 7f80157fa640 20 client.4296.objecter _get_session s=0x7f7ffc072c20 osd=3 3 2024-05-15T12:43:11.179 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.188+0000 7f80157fa640 20 client.4296.objecter get_session s=0x7f7ffc072c20 osd=3 3 2024-05-15T12:43:11.179 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.188+0000 7f80157fa640 15 client.4296.objecter _session_command_op_assign 3 1 2024-05-15T12:43:11.179 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.188+0000 7f80157fa640 20 client.4296.objecter put_session s=0x7f7ffc072c20 osd=3 4 2024-05-15T12:43:11.179 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.188+0000 7f80157fa640 10 client.4296.objecter _send_command 1 2024-05-15T12:43:11.179 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.188+0000 7f80157fa640 1 -- 172.21.15.117:0/3984485137 --> [v2:172.21.15.117:6802/3842021926,v1:172.21.15.117:6803/3842021926] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f80180a1e50 con 0x7f7ffc0732f0 2024-05-15T12:43:11.180 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.188+0000 7f80157fa640 20 client.4296.objecter dump_active .. 0 homeless 2024-05-15T12:43:11.180 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.188+0000 7f80157fa640 1 -- 172.21.15.117:0/3984485137 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f80180cf3e0 con 0x7f801002e200 2024-05-15T12:43:11.180 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.188+0000 7f80157fa640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-05-15T12:43:11.182 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.190+0000 7f801d207640 1 --2- 172.21.15.117:0/3984485137 >> [v2:172.21.15.117:6802/3842021926,v1:172.21.15.117:6803/3842021926] conn(0x7f7ffc0732f0 0x7f7ffc075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:11.182 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.191+0000 7f801d207640 10 monclient: get_auth_request con 0x7f7ffc0732f0 auth_method 0 2024-05-15T12:43:11.184 INFO:teuthology.orchestra.run.smithi117.stdout:55834574855 2024-05-15T12:43:11.184 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-05-15T12:43:11.186 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.195+0000 7f091cff9640 1 -- 172.21.15.117:0/2876865113 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f0914018020 con 0x7f09200b4d00 2024-05-15T12:43:11.186 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.195+0000 7f091cff9640 1 --2- 172.21.15.117:0/2876865113 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f090c06ce90 0x7f090c06f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.187 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.196+0000 7f6586164640 10 client.4277.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-05-15T12:43:11.187 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.196+0000 7f6586164640 15 client.4277.objecter _session_command_op_assign -1 2 2024-05-15T12:43:11.187 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.196+0000 7f6586164640 20 client.4277.objecter _get_session s=0x7f6564072c50 osd=1 3 2024-05-15T12:43:11.187 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.196+0000 7f6586164640 20 client.4277.objecter put_session s=0x7f6564072c50 osd=1 3 2024-05-15T12:43:11.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.196+0000 7f6586164640 20 client.4277.objecter _get_session s=0x7f6564072c50 osd=1 3 2024-05-15T12:43:11.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.196+0000 7f6586164640 15 client.4277.objecter _session_command_op_remove -1 2 2024-05-15T12:43:11.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.196+0000 7f6586164640 20 client.4277.objecter get_session s=0x7f6564072c50 osd=1 3 2024-05-15T12:43:11.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.196+0000 7f6586164640 15 client.4277.objecter _session_command_op_assign 1 2 2024-05-15T12:43:11.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.196+0000 7f6586164640 20 client.4277.objecter put_session s=0x7f6564072c50 osd=1 4 2024-05-15T12:43:11.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.196+0000 7f6586164640 10 client.4277.objecter _send_command 2 2024-05-15T12:43:11.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.196+0000 7f6586164640 1 -- 172.21.15.117:0/3888178903 --> [v2:172.21.15.117:6818/3864975175,v1:172.21.15.117:6819/3864975175] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6578165200 con 0x7f6564073320 2024-05-15T12:43:11.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.196+0000 7f091ffff640 1 --2- 172.21.15.117:0/2876865113 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f090c06ce90 0x7f090c06f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:11.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.196+0000 7f091ffff640 10 monclient: get_auth_request con 0x7f090c06ce90 auth_method 0 2024-05-15T12:43:11.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.197+0000 7f091ffff640 1 --2- 172.21.15.117:0/2876865113 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f090c06ce90 0x7f090c06f350 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f09200b3f10 tx=0x7f091003d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:11.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.197+0000 7f091cff9640 10 client.4283.objecter ms_handle_connect 0x7f090c06ce90 2024-05-15T12:43:11.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.197+0000 7f80157fa640 10 client.4296.objecter ms_handle_connect 0x7f7ffc0732f0 2024-05-15T12:43:11.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.197+0000 7f801d207640 1 --2- 172.21.15.117:0/3984485137 >> [v2:172.21.15.117:6802/3842021926,v1:172.21.15.117:6803/3842021926] conn(0x7f7ffc0732f0 0x7f7ffc075710 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:11.190 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.198+0000 7f38367fc640 1 -- 172.21.15.117:0/1662615750 <== osd.2 v2:172.21.15.117:6804/1425835241 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f384013efb0 con 0x7f38240732f0 2024-05-15T12:43:11.190 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.198+0000 7f38367fc640 10 client.4280.objecter ms_dispatch 0x7f3840000b90 command_reply(tid 2: 0 ) v1 2024-05-15T12:43:11.190 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.198+0000 7f38367fc640 10 client.4280.objecter _finish_command 2 = system:0 2024-05-15T12:43:11.190 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.198+0000 7f38367fc640 20 client.4280.objecter put_session s=0x7f3824072c20 osd=2 4 2024-05-15T12:43:11.191 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.198+0000 7f38367fc640 15 client.4280.objecter _session_command_op_remove 2 2 2024-05-15T12:43:11.191 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.199+0000 7f3847cbe640 10 librados: watch_flush enter 2024-05-15T12:43:11.191 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.199+0000 7f3847cbe640 10 librados: watch_flush exit 2024-05-15T12:43:11.191 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.199+0000 7f3847cbe640 10 client.4280.objecter close_session for osd.2 2024-05-15T12:43:11.191 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.199+0000 7f3847cbe640 1 -- 172.21.15.117:0/1662615750 >> [v2:172.21.15.117:6804/1425835241,v1:172.21.15.117:6805/1425835241] conn(0x7f38240732f0 msgr2=0x7f3824075710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.191 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.199+0000 7f3847cbe640 1 --2- 172.21.15.117:0/1662615750 >> [v2:172.21.15.117:6804/1425835241,v1:172.21.15.117:6805/1425835241] conn(0x7f38240732f0 0x7f3824075710 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.191 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.199+0000 7f3847cbe640 20 client.4280.objecter put_session s=0x7f3824072c20 osd=2 1 2024-05-15T12:43:11.191 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.199+0000 7f3847cbe640 20 client.4280.objecter shutdown clearing up homeless session... 2024-05-15T12:43:11.191 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.199+0000 7f3847cbe640 10 client.4280.objecter successfully canceled tick 2024-05-15T12:43:11.192 INFO:teuthology.orchestra.run.smithi117.stdout:81604378632 2024-05-15T12:43:11.192 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-05-15T12:43:11.193 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.200+0000 7f3847cbe640 1 -- 172.21.15.117:0/1662615750 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f382406ceb0 msgr2=0x7f382406f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.193 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.200+0000 7f3847cbe640 1 --2- 172.21.15.117:0/1662615750 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f382406ceb0 0x7f382406f370 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f383c036000 tx=0x7f383c030000 comp rx=0 tx=0).stop 2024-05-15T12:43:11.193 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.200+0000 7f3847cbe640 10 monclient: shutdown 2024-05-15T12:43:11.193 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.200+0000 7f3847cbe640 20 monclient: shutdown canceling and discarding version request 1 2024-05-15T12:43:11.193 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.200+0000 7f3847cbe640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:11.193 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.200+0000 7f3847cbe640 1 -- 172.21.15.117:0/1662615750 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f384013dac0 msgr2=0x7f384013ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.193 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.200+0000 7f3847cbe640 1 --2- 172.21.15.117:0/1662615750 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f384013dac0 0x7f384013ea70 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f383800ba10 tx=0x7f383800bed0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.193 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.200+0000 7f384625b640 1 -- 172.21.15.117:0/1662615750 reap_dead start 2024-05-15T12:43:11.193 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.200+0000 7f3847cbe640 1 -- 172.21.15.117:0/1662615750 shutdown_connections 2024-05-15T12:43:11.193 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.200+0000 7f3847cbe640 1 -- 172.21.15.117:0/1662615750 >> 172.21.15.117:0/1662615750 conn(0x7f3840066cc0 msgr2=0x7f3840074610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:11.193 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.200+0000 7f3847cbe640 1 -- 172.21.15.117:0/1662615750 shutdown_connections 2024-05-15T12:43:11.193 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.201+0000 7f3847cbe640 1 -- 172.21.15.117:0/1662615750 wait complete. 2024-05-15T12:43:11.194 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.201+0000 7f3847cbe640 1 librados: shutdown 2024-05-15T12:43:11.194 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.204+0000 7f091cff9640 1 -- 172.21.15.117:0/2876865113 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f0914057080 con 0x7f09200b4d00 2024-05-15T12:43:11.194 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.204+0000 7f091cff9640 10 client.4283.objecter ms_dispatch 0x7f0920000b90 osd_map(26..26 src has 1..26) v4 2024-05-15T12:43:11.194 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.204+0000 7f091cff9640 3 client.4283.objecter handle_osd_map got epochs [26,26] > 0 2024-05-15T12:43:11.194 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.204+0000 7f091cff9640 3 client.4283.objecter handle_osd_map decoding full epoch 26 2024-05-15T12:43:11.195 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.204+0000 7f091cff9640 10 client.4283.objecter checking command 1 2024-05-15T12:43:11.195 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.204+0000 7f091cff9640 1 --2- 172.21.15.117:0/2876865113 >> [v2:172.21.15.178:6816/3713764498,v1:172.21.15.178:6817/3713764498] conn(0x7f090c0732d0 0x7f090c0756f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.195 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.204+0000 7f091cff9640 20 client.4283.objecter _get_session s=0x7f090c072c00 osd=5 3 2024-05-15T12:43:11.195 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.204+0000 7f091cff9640 20 client.4283.objecter put_session s=0x7f090c072c00 osd=5 3 2024-05-15T12:43:11.195 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.204+0000 7f091cff9640 15 client.4283.objecter _session_command_op_remove -1 1 2024-05-15T12:43:11.195 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.204+0000 7f091cff9640 20 client.4283.objecter _get_session s=0x7f090c072c00 osd=5 3 2024-05-15T12:43:11.195 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.204+0000 7f091cff9640 20 client.4283.objecter get_session s=0x7f090c072c00 osd=5 3 2024-05-15T12:43:11.195 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.204+0000 7f091cff9640 15 client.4283.objecter _session_command_op_assign 5 1 2024-05-15T12:43:11.195 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.204+0000 7f091cff9640 20 client.4283.objecter put_session s=0x7f090c072c00 osd=5 4 2024-05-15T12:43:11.195 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.204+0000 7f091cff9640 10 client.4283.objecter _send_command 1 2024-05-15T12:43:11.195 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.204+0000 7f091cff9640 1 -- 172.21.15.117:0/2876865113 --> [v2:172.21.15.178:6816/3713764498,v1:172.21.15.178:6817/3713764498] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0914006780 con 0x7f090c0732d0 2024-05-15T12:43:11.195 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.204+0000 7f091cff9640 20 client.4283.objecter dump_active .. 0 homeless 2024-05-15T12:43:11.196 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.205+0000 7f0924c8c640 1 --2- 172.21.15.117:0/2876865113 >> [v2:172.21.15.178:6816/3713764498,v1:172.21.15.178:6817/3713764498] conn(0x7f090c0732d0 0x7f090c0756f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:11.196 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.205+0000 7f0924c8c640 10 monclient: get_auth_request con 0x7f090c0732d0 auth_method 0 2024-05-15T12:43:11.198 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.208+0000 7f0924c8c640 1 --2- 172.21.15.117:0/2876865113 >> [v2:172.21.15.178:6816/3713764498,v1:172.21.15.178:6817/3713764498] conn(0x7f090c0732d0 0x7f090c0756f0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:11.198 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.208+0000 7f091cff9640 10 client.4283.objecter ms_handle_connect 0x7f090c0732d0 2024-05-15T12:43:11.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.213+0000 7f656bfff640 1 -- 172.21.15.117:0/3888178903 <== osd.1 v2:172.21.15.117:6818/3864975175 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f656c030850 con 0x7f6564073320 2024-05-15T12:43:11.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.213+0000 7f656bfff640 10 client.4277.objecter ms_dispatch 0x7f6580000b90 command_reply(tid 2: 0 ) v1 2024-05-15T12:43:11.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.213+0000 7f656bfff640 10 client.4277.objecter _finish_command 2 = system:0 2024-05-15T12:43:11.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.213+0000 7f656bfff640 20 client.4277.objecter put_session s=0x7f6564072c50 osd=1 4 2024-05-15T12:43:11.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.213+0000 7f656bfff640 15 client.4277.objecter _session_command_op_remove 1 2 2024-05-15T12:43:11.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.214+0000 7f6586164640 10 librados: watch_flush enter 2024-05-15T12:43:11.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.214+0000 7f6586164640 10 librados: watch_flush exit 2024-05-15T12:43:11.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.214+0000 7f6586164640 10 client.4277.objecter close_session for osd.1 2024-05-15T12:43:11.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.214+0000 7f6586164640 1 -- 172.21.15.117:0/3888178903 >> [v2:172.21.15.117:6818/3864975175,v1:172.21.15.117:6819/3864975175] conn(0x7f6564073320 msgr2=0x7f6564075740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.214+0000 7f6586164640 1 --2- 172.21.15.117:0/3888178903 >> [v2:172.21.15.117:6818/3864975175,v1:172.21.15.117:6819/3864975175] conn(0x7f6564073320 0x7f6564075740 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.214+0000 7f6586164640 20 client.4277.objecter put_session s=0x7f6564072c50 osd=1 1 2024-05-15T12:43:11.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.214+0000 7f6586164640 20 client.4277.objecter shutdown clearing up homeless session... 2024-05-15T12:43:11.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.214+0000 7f6586164640 10 client.4277.objecter successfully canceled tick 2024-05-15T12:43:11.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.214+0000 7f6586164640 1 -- 172.21.15.117:0/3888178903 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f656406cee0 msgr2=0x7f656406f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.214+0000 7f6586164640 1 --2- 172.21.15.117:0/3888178903 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f656406cee0 0x7f656406f3a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6580068f90 tx=0x7f6580069630 comp rx=0 tx=0).stop 2024-05-15T12:43:11.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.214+0000 7f6586164640 10 monclient: shutdown 2024-05-15T12:43:11.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.214+0000 7f6586164640 20 monclient: shutdown canceling and discarding version request 1 2024-05-15T12:43:11.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.214+0000 7f6586164640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:11.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.214+0000 7f6586164640 1 -- 172.21.15.117:0/3888178903 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6578033590 msgr2=0x7f657802eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.214+0000 7f6586164640 1 --2- 172.21.15.117:0/3888178903 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6578033590 0x7f657802eab0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f657400b410 tx=0x7f657400b8d0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.214+0000 7f657ffff640 1 -- 172.21.15.117:0/3888178903 reap_dead start 2024-05-15T12:43:11.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.214+0000 7f6586164640 1 -- 172.21.15.117:0/3888178903 shutdown_connections 2024-05-15T12:43:11.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.214+0000 7f6586164640 1 -- 172.21.15.117:0/3888178903 >> 172.21.15.117:0/3888178903 conn(0x7f65780061d0 msgr2=0x7f65780ce7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:11.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.214+0000 7f6586164640 1 -- 172.21.15.117:0/3888178903 shutdown_connections 2024-05-15T12:43:11.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.214+0000 7f6586164640 1 -- 172.21.15.117:0/3888178903 wait complete. 2024-05-15T12:43:11.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.215+0000 7f6586164640 1 librados: shutdown 2024-05-15T12:43:11.210 INFO:teuthology.orchestra.run.smithi117.stdout:85899345930 2024-05-15T12:43:11.211 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-05-15T12:43:11.213 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.223+0000 7f091cff9640 1 -- 172.21.15.117:0/2876865113 <== mon.2 v2:172.21.15.117:3301/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0914055280 con 0x7f09200b4d00 2024-05-15T12:43:11.214 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.223+0000 7f091cff9640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-05-15T12:43:11.222 INFO:teuthology.orchestra.run.smithi117.stdout:85899345929 2024-05-15T12:43:11.223 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-05-15T12:43:11.239 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.249+0000 7f091cff9640 1 -- 172.21.15.117:0/2876865113 <== osd.5 v2:172.21.15.178:6816/3713764498 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f0914006780 con 0x7f090c0732d0 2024-05-15T12:43:11.239 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.249+0000 7f091cff9640 10 client.4283.objecter ms_dispatch 0x7f0920000b90 command_reply(tid 1: 0 ) v1 2024-05-15T12:43:11.239 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.249+0000 7f091cff9640 10 client.4283.objecter _finish_command 1 = system:0 2024-05-15T12:43:11.239 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.249+0000 7f091cff9640 20 client.4283.objecter put_session s=0x7f090c072c00 osd=5 4 2024-05-15T12:43:11.239 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.249+0000 7f091cff9640 15 client.4283.objecter _session_command_op_remove 5 1 2024-05-15T12:43:11.272 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.278+0000 7f80157fa640 1 -- 172.21.15.117:0/3984485137 <== osd.3 v2:172.21.15.117:6802/3842021926 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f80180a1e50 con 0x7f7ffc0732f0 2024-05-15T12:43:11.273 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.278+0000 7f80157fa640 10 client.4296.objecter ms_dispatch 0x7f8018000b90 command_reply(tid 1: 0 ) v1 2024-05-15T12:43:11.273 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.278+0000 7f80157fa640 10 client.4296.objecter _finish_command 1 = system:0 2024-05-15T12:43:11.273 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.278+0000 7f80157fa640 20 client.4296.objecter put_session s=0x7f7ffc072c20 osd=3 4 2024-05-15T12:43:11.273 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.278+0000 7f80157fa640 15 client.4296.objecter _session_command_op_remove 3 1 2024-05-15T12:43:11.277 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.286+0000 7f09266ef640 10 client.4283.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-05-15T12:43:11.277 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.286+0000 7f09266ef640 15 client.4283.objecter _session_command_op_assign -1 2 2024-05-15T12:43:11.277 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.286+0000 7f09266ef640 20 client.4283.objecter _get_session s=0x7f090c072c00 osd=5 3 2024-05-15T12:43:11.277 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.286+0000 7f09266ef640 20 client.4283.objecter put_session s=0x7f090c072c00 osd=5 3 2024-05-15T12:43:11.277 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.286+0000 7f09266ef640 20 client.4283.objecter _get_session s=0x7f090c072c00 osd=5 3 2024-05-15T12:43:11.277 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.286+0000 7f09266ef640 15 client.4283.objecter _session_command_op_remove -1 2 2024-05-15T12:43:11.277 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.286+0000 7f09266ef640 20 client.4283.objecter get_session s=0x7f090c072c00 osd=5 3 2024-05-15T12:43:11.277 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.286+0000 7f09266ef640 15 client.4283.objecter _session_command_op_assign 5 2 2024-05-15T12:43:11.277 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.286+0000 7f09266ef640 20 client.4283.objecter put_session s=0x7f090c072c00 osd=5 4 2024-05-15T12:43:11.277 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.286+0000 7f09266ef640 10 client.4283.objecter _send_command 2 2024-05-15T12:43:11.277 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.286+0000 7f09266ef640 1 -- 172.21.15.117:0/2876865113 --> [v2:172.21.15.178:6816/3713764498,v1:172.21.15.178:6817/3713764498] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f09200b75d0 con 0x7f090c0732d0 2024-05-15T12:43:11.282 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.291+0000 7f9ef359e640 10 monclient: get_monmap_and_config 2024-05-15T12:43:11.282 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.291+0000 7f9ef359e640 10 monclient: build_initial_monmap 2024-05-15T12:43:11.282 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.291+0000 7f9ef359e640 10 monclient: monmap: 2024-05-15T12:43:11.282 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:11.282 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:11.282 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:43:11.293010+0000 2024-05-15T12:43:11.282 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:43:11.293010+0000 2024-05-15T12:43:11.282 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:11.282 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.282 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T12:43:11.282 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T12:43:11.283 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T12:43:11.283 INFO:teuthology.orchestra.run.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T12:43:11.283 INFO:teuthology.orchestra.run.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T12:43:11.283 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.283 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.292+0000 7f091cff9640 1 -- 172.21.15.117:0/2876865113 <== osd.5 v2:172.21.15.178:6816/3713764498 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f09200b75d0 con 0x7f090c0732d0 2024-05-15T12:43:11.283 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.292+0000 7f091cff9640 10 client.4283.objecter ms_dispatch 0x7f0920000b90 command_reply(tid 2: 0 ) v1 2024-05-15T12:43:11.283 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.292+0000 7f091cff9640 10 client.4283.objecter _finish_command 2 = system:0 2024-05-15T12:43:11.283 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.292+0000 7f091cff9640 20 client.4283.objecter put_session s=0x7f090c072c00 osd=5 4 2024-05-15T12:43:11.283 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.292+0000 7f091cff9640 15 client.4283.objecter _session_command_op_remove 5 2 2024-05-15T12:43:11.284 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.293+0000 7f9ef359e640 1 Processor -- start 2024-05-15T12:43:11.284 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.293+0000 7f9ef359e640 1 -- start start 2024-05-15T12:43:11.284 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.293+0000 7f9ef359e640 10 monclient: init 2024-05-15T12:43:11.286 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.293+0000 7f0906ffd640 10 librados: watch_flush enter 2024-05-15T12:43:11.286 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.293+0000 7f0906ffd640 10 librados: watch_flush exit 2024-05-15T12:43:11.286 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.293+0000 7f0906ffd640 10 client.4283.objecter close_session for osd.5 2024-05-15T12:43:11.286 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.293+0000 7f0906ffd640 1 -- 172.21.15.117:0/2876865113 >> [v2:172.21.15.178:6816/3713764498,v1:172.21.15.178:6817/3713764498] conn(0x7f090c0732d0 msgr2=0x7f090c0756f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.286 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.293+0000 7f0906ffd640 1 --2- 172.21.15.117:0/2876865113 >> [v2:172.21.15.178:6816/3713764498,v1:172.21.15.178:6817/3713764498] conn(0x7f090c0732d0 0x7f090c0756f0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.286 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.293+0000 7f0906ffd640 20 client.4283.objecter put_session s=0x7f090c072c00 osd=5 1 2024-05-15T12:43:11.286 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.293+0000 7f0906ffd640 20 client.4283.objecter shutdown clearing up homeless session... 2024-05-15T12:43:11.287 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.293+0000 7f0906ffd640 10 client.4283.objecter successfully canceled tick 2024-05-15T12:43:11.287 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.294+0000 7f0906ffd640 1 -- 172.21.15.117:0/2876865113 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f090c06ce90 msgr2=0x7f090c06f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.287 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.294+0000 7f0906ffd640 1 --2- 172.21.15.117:0/2876865113 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f090c06ce90 0x7f090c06f350 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f09200b3f10 tx=0x7f091003d040 comp rx=0 tx=0).stop 2024-05-15T12:43:11.287 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.294+0000 7f0906ffd640 10 monclient: shutdown 2024-05-15T12:43:11.287 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.294+0000 7f0906ffd640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:11.287 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.294+0000 7f0906ffd640 1 -- 172.21.15.117:0/2876865113 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f09200b4d00 msgr2=0x7f09201db4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.287 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.294+0000 7f0906ffd640 1 --2- 172.21.15.117:0/2876865113 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f09200b4d00 0x7f09201db4b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f091400b440 tx=0x7f091400b900 comp rx=0 tx=0).stop 2024-05-15T12:43:11.287 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.294+0000 7f0924c8c640 1 -- 172.21.15.117:0/2876865113 reap_dead start 2024-05-15T12:43:11.287 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.294+0000 7f0906ffd640 1 -- 172.21.15.117:0/2876865113 shutdown_connections 2024-05-15T12:43:11.287 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.294+0000 7f0906ffd640 1 -- 172.21.15.117:0/2876865113 >> 172.21.15.117:0/2876865113 conn(0x7f0920066cc0 msgr2=0x7f09200b67e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:11.288 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.294+0000 7f9ef359e640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:11.288 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.294+0000 7f9ef359e640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-05-15T12:43:11.289 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.294+0000 7f9ef359e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9eec006fd0 addr v1:172.21.15.117:6789/0 2024-05-15T12:43:11.289 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.295+0000 7f9ef359e640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f9eec0073b0 0x7f9eec0cb150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.289 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.295+0000 7f9ef359e640 10 monclient(hunting): picked mon.noname-a con 0x7f9eec0073b0 addr v2:172.21.15.117:3300/0 2024-05-15T12:43:11.289 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.295+0000 7f9ef359e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9eec033930 addr v1:172.21.15.178:6789/0 2024-05-15T12:43:11.289 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.295+0000 7f9ef359e640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9eec037050 con 0x7f9eec006fd0 2024-05-15T12:43:11.289 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.295+0000 7f9ef359e640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9eec037330 con 0x7f9eec033930 2024-05-15T12:43:11.289 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.295+0000 7f9ef359e640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:11.289 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.295+0000 7f9ef359e640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9eec0357e0 con 0x7f9eec0073b0 2024-05-15T12:43:11.289 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.295+0000 7f9ef359e640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:11.289 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.295+0000 7f9ef359e640 10 monclient(hunting): authenticate will time out at 1354.537487s 2024-05-15T12:43:11.290 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.295+0000 7f0906ffd640 1 -- 172.21.15.117:0/2876865113 shutdown_connections 2024-05-15T12:43:11.290 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.296+0000 7f0906ffd640 1 -- 172.21.15.117:0/2876865113 wait complete. 2024-05-15T12:43:11.290 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.296+0000 7f0906ffd640 1 librados: shutdown 2024-05-15T12:43:11.290 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.296+0000 7f9ef259c640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f9eec006fd0 0x7f9eec035f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:48670/0 (socket says 172.21.15.117:48670) 2024-05-15T12:43:11.290 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.296+0000 7f9ef259c640 1 -- 172.21.15.117:0/357197860 learned_addr learned my addr 172.21.15.117:0/357197860 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-05-15T12:43:11.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.302+0000 7f9ef1d9b640 1 --2- 172.21.15.117:0/357197860 >> v2:172.21.15.117:3300/0 conn(0x7f9eec0073b0 0x7f9eec0cb150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:11.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.302+0000 7f9ef1d9b640 10 monclient(hunting): get_auth_request con 0x7f9eec0073b0 auth_method 0 2024-05-15T12:43:11.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.302+0000 7f9ef1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:11.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.302+0000 7f9ef1d9b640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:11.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.302+0000 7f9ef1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:11.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.302+0000 7f9ef159a640 1 -- 172.21.15.117:0/357197860 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 926965853 0 0) 0x7f9eec037050 con 0x7f9eec006fd0 2024-05-15T12:43:11.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.302+0000 7f9ef159a640 10 monclient(hunting): _negotiate 2024-05-15T12:43:11.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.302+0000 7f9ef159a640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:11.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.302+0000 7f9ef159a640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:11.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.302+0000 7f9ef159a640 10 monclient(hunting): my global_id is 4299 2024-05-15T12:43:11.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.302+0000 7f9ef159a640 1 -- 172.21.15.117:0/357197860 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9edc003520 con 0x7f9eec006fd0 2024-05-15T12:43:11.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.303+0000 7f9ef1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:11.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.303+0000 7f9ef1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:11.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.303+0000 7f9ef1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:11.301 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.310+0000 7f9ef1d9b640 10 monclient(hunting): handle_auth_done global_id 4302 payload 274 2024-05-15T12:43:11.301 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.310+0000 7f9ef1d9b640 1 -- 172.21.15.117:0/357197860 >> v1:172.21.15.178:6789/0 conn(0x7f9eec033930 legacy=0x7f9eec0d24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.301 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.310+0000 7f9ef1d9b640 1 -- 172.21.15.117:0/357197860 >> v1:172.21.15.117:6789/0 conn(0x7f9eec006fd0 legacy=0x7f9eec035f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.310+0000 7f9ef1d9b640 10 monclient: _finish_hunting 0 2024-05-15T12:43:11.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.310+0000 7f9ef1d9b640 1 monclient: found mon.noname-a 2024-05-15T12:43:11.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.310+0000 7f9ef1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:11.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.310+0000 7f9ef1d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.117:3300/0 2024-05-15T12:43:11.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.310+0000 7f9ef1d9b640 1 -- 172.21.15.117:0/357197860 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eec037610 con 0x7f9eec0073b0 2024-05-15T12:43:11.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.311+0000 7f801da08640 10 client.4296.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-05-15T12:43:11.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.311+0000 7f801da08640 15 client.4296.objecter _session_command_op_assign -1 2 2024-05-15T12:43:11.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.311+0000 7f801da08640 20 client.4296.objecter _get_session s=0x7f7ffc072c20 osd=3 3 2024-05-15T12:43:11.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.311+0000 7f801da08640 20 client.4296.objecter put_session s=0x7f7ffc072c20 osd=3 3 2024-05-15T12:43:11.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.311+0000 7f801da08640 20 client.4296.objecter _get_session s=0x7f7ffc072c20 osd=3 3 2024-05-15T12:43:11.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.311+0000 7f801da08640 15 client.4296.objecter _session_command_op_remove -1 2 2024-05-15T12:43:11.304 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.311+0000 7f801da08640 20 client.4296.objecter get_session s=0x7f7ffc072c20 osd=3 3 2024-05-15T12:43:11.304 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.311+0000 7f801da08640 15 client.4296.objecter _session_command_op_assign 3 2 2024-05-15T12:43:11.304 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.311+0000 7f801da08640 20 client.4296.objecter put_session s=0x7f7ffc072c20 osd=3 4 2024-05-15T12:43:11.304 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.311+0000 7f801da08640 10 client.4296.objecter _send_command 2 2024-05-15T12:43:11.304 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.311+0000 7f801da08640 1 -- 172.21.15.117:0/3984485137 --> [v2:172.21.15.117:6802/3842021926,v1:172.21.15.117:6803/3842021926] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8010005390 con 0x7f7ffc0732f0 2024-05-15T12:43:11.305 INFO:teuthology.orchestra.run.smithi117.stdout:60129542151 2024-05-15T12:43:11.305 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-05-15T12:43:11.308 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.318+0000 7f9ef1d9b640 1 --2- 172.21.15.117:0/357197860 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9eec0073b0 0x7f9eec0cb150 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9ee80097a0 tx=0x7f9ee802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3bd2a0e6136ab50 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:11.316 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.324+0000 7f9ef159a640 1 -- 172.21.15.117:0/357197860 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ee8035020 con 0x7f9eec0073b0 2024-05-15T12:43:11.316 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.324+0000 7f9ef159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:11.316 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.324+0000 7f9ef159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T12:43:11.316 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.324+0000 7f9ef159a640 10 monclient: dump: 2024-05-15T12:43:11.316 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:11.316 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:11.316 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.316 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.316 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:11.316 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.317 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:11.317 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:11.317 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:11.317 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.324+0000 7f9ef159a640 10 monclient: _finish_auth 0 2024-05-15T12:43:11.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.324+0000 7f9ef159a640 10 monclient: _check_auth_tickets 2024-05-15T12:43:11.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.324+0000 7f9ef159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:11.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.324+0000 7f9ef159a640 1 -- 172.21.15.117:0/357197860 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ee8031870 con 0x7f9eec0073b0 2024-05-15T12:43:11.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.324+0000 7f9ef159a640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:11.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.324+0000 7f9ef159a640 1 -- 172.21.15.117:0/357197860 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ee8031c80 con 0x7f9eec0073b0 2024-05-15T12:43:11.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.324+0000 7f9ef159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:11.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.324+0000 7f9ef159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T12:43:11.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.324+0000 7f9ef159a640 10 monclient: dump: 2024-05-15T12:43:11.317 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:11.317 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:11.317 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.317 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.318 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:11.318 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.318 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:11.318 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:11.318 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:11.318 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.324+0000 7f9ef359e640 5 monclient: authenticate success, global_id 4302 2024-05-15T12:43:11.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.324+0000 7f9ef359e640 10 monclient: get_monmap_and_config success 2024-05-15T12:43:11.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.324+0000 7f9ef359e640 4 set_mon_vals no callback set 2024-05-15T12:43:11.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.325+0000 7f9ef359e640 10 monclient: shutdown 2024-05-15T12:43:11.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.326+0000 7f9ef359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:11.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.326+0000 7f9ef359e640 1 -- 172.21.15.117:0/357197860 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9eec0073b0 msgr2=0x7f9eec0cb150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.326+0000 7f9ef359e640 1 --2- 172.21.15.117:0/357197860 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9eec0073b0 0x7f9eec0cb150 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9ee80097a0 tx=0x7f9ee802b910 comp rx=0 tx=0).stop 2024-05-15T12:43:11.319 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.328+0000 7f9ef359e640 1 -- 172.21.15.117:0/357197860 shutdown_connections 2024-05-15T12:43:11.319 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.329+0000 7f9ef359e640 1 --2- 172.21.15.117:0/357197860 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9eec0073b0 0x7f9eec0cb150 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.319 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.329+0000 7f9ef359e640 1 -- 172.21.15.117:0/357197860 >> 172.21.15.117:0/357197860 conn(0x7f9eec0061a0 msgr2=0x7f9eec0065b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:11.319 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.329+0000 7f9ef359e640 1 -- 172.21.15.117:0/357197860 shutdown_connections 2024-05-15T12:43:11.319 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.329+0000 7f9ef359e640 1 -- 172.21.15.117:0/357197860 wait complete. 2024-05-15T12:43:11.321 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.330+0000 7f9ef359e640 10 monclient: build_initial_monmap 2024-05-15T12:43:11.321 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.331+0000 7f9ef359e640 10 monclient: monmap: 2024-05-15T12:43:11.321 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:11.321 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:43:11.321 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-05-15T12:43:11.321 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-05-15T12:43:11.321 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:11.321 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.321 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T12:43:11.321 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T12:43:11.321 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T12:43:11.321 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.322 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.331+0000 7f9ef359e640 1 librados: starting msgr at 2024-05-15T12:43:11.322 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.331+0000 7f9ef359e640 1 librados: starting objecter 2024-05-15T12:43:11.322 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.331+0000 7f9ef359e640 1 Processor -- start 2024-05-15T12:43:11.323 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.332+0000 7f9ef359e640 1 -- start start 2024-05-15T12:43:11.323 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.332+0000 7f9ef359e640 1 librados: setting wanted keys 2024-05-15T12:43:11.323 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.332+0000 7f9ef359e640 1 librados: calling monclient init 2024-05-15T12:43:11.323 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.332+0000 7f9ef359e640 10 monclient: init 2024-05-15T12:43:11.326 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.333+0000 7f9ef359e640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:11.326 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.333+0000 7f9ef359e640 10 monclient: _add_conns ranks=[2,1,0] 2024-05-15T12:43:11.326 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.333+0000 7f9ef359e640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9eec006fd0 0x7f9eec160210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.326 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.333+0000 7f9ef359e640 10 monclient(hunting): picked mon.noname-c con 0x7f9eec006fd0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:43:11.326 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.333+0000 7f9ef359e640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9eec033930 0x7f9eec0c1570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.326 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.333+0000 7f9ef359e640 10 monclient(hunting): picked mon.noname-b con 0x7f9eec033930 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:43:11.326 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.333+0000 7f9ef359e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9eec160c30 0x7f9eec0c1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.326 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.333+0000 7f9ef359e640 10 monclient(hunting): picked mon.noname-a con 0x7f9eec160c30 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T12:43:11.326 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.334+0000 7f9ef359e640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:11.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.334+0000 7f9ef359e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eec033d80 con 0x7f9eec160c30 2024-05-15T12:43:11.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.334+0000 7f9ef359e640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:11.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.334+0000 7f9ef359e640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eec035ac0 con 0x7f9eec033930 2024-05-15T12:43:11.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.334+0000 7f9ef359e640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:11.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.334+0000 7f9ef1d9b640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9eec033930 0x7f9eec0c1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:11.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.334+0000 7f9ef359e640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9eec033f00 con 0x7f9eec006fd0 2024-05-15T12:43:11.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.334+0000 7f9ef359e640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:11.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.334+0000 7f9ef1d9b640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9eec033930 0x7f9eec0c1570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:42476/0 (socket says 172.21.15.117:42476) 2024-05-15T12:43:11.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.334+0000 7f9ef1d9b640 1 -- 172.21.15.117:0/723854573 learned_addr learned my addr 172.21.15.117:0/723854573 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:43:11.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.334+0000 7f9ef359e640 10 monclient(hunting): authenticate will time out at 1354.576361s 2024-05-15T12:43:11.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.334+0000 7f9ef259c640 1 --2- 172.21.15.117:0/723854573 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9eec006fd0 0x7f9eec160210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:11.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.334+0000 7f9ef259c640 10 monclient(hunting): get_auth_request con 0x7f9eec006fd0 auth_method 0 2024-05-15T12:43:11.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.334+0000 7f9ef259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:11.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.334+0000 7f9ef259c640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:11.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.334+0000 7f9ef259c640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:11.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.335+0000 7f9ef1d9b640 10 monclient(hunting): get_auth_request con 0x7f9eec033930 auth_method 0 2024-05-15T12:43:11.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.335+0000 7f9ef1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:11.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.335+0000 7f9ef1d9b640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:11.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.335+0000 7f9ef1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:11.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.335+0000 7f9ef259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:11.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.335+0000 7f9ef259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:11.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.335+0000 7f9ef259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:11.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.338+0000 7f9ef1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:11.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.338+0000 7f9ef1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:11.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.338+0000 7f9ef1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:11.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.339+0000 7f9ef2d9d640 1 --2- 172.21.15.117:0/723854573 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9eec160c30 0x7f9eec0c1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:11.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.339+0000 7f9ef2d9d640 10 monclient(hunting): get_auth_request con 0x7f9eec160c30 auth_method 0 2024-05-15T12:43:11.330 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.339+0000 7f9ef2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:11.330 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.339+0000 7f9ef2d9d640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:11.330 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.339+0000 7f9ef2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:11.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.341+0000 7f9ef2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:11.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.341+0000 7f9ef2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:11.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.341+0000 7f9ef2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:11.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.342+0000 7f9ef1d9b640 10 monclient(hunting): handle_auth_done global_id 4348 payload 824 2024-05-15T12:43:11.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.342+0000 7f9ef1d9b640 1 -- 172.21.15.117:0/723854573 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9eec006fd0 msgr2=0x7f9eec160210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.342+0000 7f9ef1d9b640 1 --2- 172.21.15.117:0/723854573 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9eec006fd0 0x7f9eec160210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.342+0000 7f9ef1d9b640 1 -- 172.21.15.117:0/723854573 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9eec160c30 msgr2=0x7f9eec0c1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.342+0000 7f9ef1d9b640 1 --2- 172.21.15.117:0/723854573 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9eec160c30 0x7f9eec0c1ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.342+0000 7f9ef1d9b640 10 monclient: _finish_hunting 0 2024-05-15T12:43:11.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.342+0000 7f9ef1d9b640 1 monclient: found mon.noname-b 2024-05-15T12:43:11.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.342+0000 7f9ef1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:11.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.342+0000 7f9ef1d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2024-05-15T12:43:11.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.342+0000 7f9ef1d9b640 1 -- 172.21.15.117:0/723854573 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eec0c8310 con 0x7f9eec033930 2024-05-15T12:43:11.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.342+0000 7f9ef259c640 1 --2- 172.21.15.117:0/723854573 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9eec006fd0 0x7f9eec160210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-15T12:43:11.338 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.348+0000 7f9ef1d9b640 1 --2- 172.21.15.117:0/723854573 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9eec033930 0x7f9eec0c1570 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9ee80098c0 tx=0x7f9ee80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:11.338 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.348+0000 7f9ee2ffd640 10 client.?.objecter ms_handle_connect 0x7f9eec033930 2024-05-15T12:43:11.338 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.348+0000 7f9ee2ffd640 10 client.?.objecter resend_mon_ops 2024-05-15T12:43:11.340 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.350+0000 7f80157fa640 1 -- 172.21.15.117:0/3984485137 <== osd.3 v2:172.21.15.117:6802/3842021926 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8010005390 con 0x7f7ffc0732f0 2024-05-15T12:43:11.340 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.350+0000 7f80157fa640 10 client.4296.objecter ms_dispatch 0x7f8018000b90 command_reply(tid 2: 0 ) v1 2024-05-15T12:43:11.341 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.350+0000 7f80157fa640 10 client.4296.objecter _finish_command 2 = system:0 2024-05-15T12:43:11.341 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.350+0000 7f80157fa640 20 client.4296.objecter put_session s=0x7f7ffc072c20 osd=3 4 2024-05-15T12:43:11.341 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.350+0000 7f80157fa640 15 client.4296.objecter _session_command_op_remove 3 2 2024-05-15T12:43:11.341 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.351+0000 7f9ee2ffd640 1 -- 172.21.15.117:0/723854573 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ee8041070 con 0x7f9eec033930 2024-05-15T12:43:11.342 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.351+0000 7f9ee2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:11.342 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.351+0000 7f9ee2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:43:11.342 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.351+0000 7f9ee2ffd640 10 monclient: dump: 2024-05-15T12:43:11.342 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:11.342 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:11.342 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.342 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.342 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:11.342 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.342 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:11.342 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:11.342 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:11.342 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.343 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.351+0000 7f9ee2ffd640 10 monclient: _finish_auth 0 2024-05-15T12:43:11.343 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.351+0000 7f9ee2ffd640 10 monclient: _check_auth_tickets 2024-05-15T12:43:11.343 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.351+0000 7f9ee2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:11.343 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.351+0000 7f9ee2ffd640 1 -- 172.21.15.117:0/723854573 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ee803b430 con 0x7f9eec033930 2024-05-15T12:43:11.343 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.351+0000 7f9ee2ffd640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:11.343 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.351+0000 7f8002ffd640 10 librados: watch_flush enter 2024-05-15T12:43:11.343 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.351+0000 7f8002ffd640 10 librados: watch_flush exit 2024-05-15T12:43:11.343 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.351+0000 7f8002ffd640 10 client.4296.objecter close_session for osd.3 2024-05-15T12:43:11.343 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.351+0000 7f8002ffd640 1 -- 172.21.15.117:0/3984485137 >> [v2:172.21.15.117:6802/3842021926,v1:172.21.15.117:6803/3842021926] conn(0x7f7ffc0732f0 msgr2=0x7f7ffc075710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.343 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.351+0000 7f8002ffd640 1 --2- 172.21.15.117:0/3984485137 >> [v2:172.21.15.117:6802/3842021926,v1:172.21.15.117:6803/3842021926] conn(0x7f7ffc0732f0 0x7f7ffc075710 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.343 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.351+0000 7f8002ffd640 20 client.4296.objecter put_session s=0x7f7ffc072c20 osd=3 1 2024-05-15T12:43:11.343 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.351+0000 7f8002ffd640 20 client.4296.objecter shutdown clearing up homeless session... 2024-05-15T12:43:11.343 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.351+0000 7f8002ffd640 10 client.4296.objecter successfully canceled tick 2024-05-15T12:43:11.344 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.351+0000 7f9ef0d99640 4 set_mon_vals no callback set 2024-05-15T12:43:11.344 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.353+0000 7f9ee2ffd640 1 -- 172.21.15.117:0/723854573 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ee8035040 con 0x7f9eec033930 2024-05-15T12:43:11.344 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.353+0000 7f9ee2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:11.344 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.353+0000 7f9ee2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:43:11.344 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.353+0000 7f9ee2ffd640 10 monclient: dump: 2024-05-15T12:43:11.344 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:11.344 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:11.344 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.344 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.345 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:11.345 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.345 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:11.345 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:11.345 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:11.345 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.345 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.353+0000 7f9ef359e640 5 monclient: authenticate success, global_id 4348 2024-05-15T12:43:11.345 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.353+0000 7f9ef359e640 10 monclient: _renew_subs 2024-05-15T12:43:11.345 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.353+0000 7f9ef359e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:43:11.345 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.353+0000 7f9ef359e640 1 -- 172.21.15.117:0/723854573 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9eec0cad90 con 0x7f9eec033930 2024-05-15T12:43:11.345 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.353+0000 7f9ef359e640 10 client.4348.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:43:11.345 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.353+0000 7f9ef359e640 10 monclient: _renew_subs 2024-05-15T12:43:11.345 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.353+0000 7f9ef359e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:43:11.345 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.353+0000 7f9ef359e640 1 -- 172.21.15.117:0/723854573 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9eec0c2280 con 0x7f9eec033930 2024-05-15T12:43:11.345 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.353+0000 7f9ef359e640 1 librados: init done 2024-05-15T12:43:11.346 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.352+0000 7f8002ffd640 1 -- 172.21.15.117:0/3984485137 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f7ffc06ceb0 msgr2=0x7f7ffc06f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.346 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.352+0000 7f8002ffd640 1 --2- 172.21.15.117:0/3984485137 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f7ffc06ceb0 0x7f7ffc06f370 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8004036000 tx=0x7f8004030000 comp rx=0 tx=0).stop 2024-05-15T12:43:11.346 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.352+0000 7f8002ffd640 10 monclient: shutdown 2024-05-15T12:43:11.346 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.352+0000 7f8002ffd640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:11.346 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.352+0000 7f8002ffd640 1 -- 172.21.15.117:0/3984485137 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f801002e200 msgr2=0x7f80100a3c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.346 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.353+0000 7f8002ffd640 1 --2- 172.21.15.117:0/3984485137 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f801002e200 0x7f80100a3c80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f801807adc0 tx=0x7f8018068a00 comp rx=0 tx=0).stop 2024-05-15T12:43:11.346 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.353+0000 7f801d207640 1 -- 172.21.15.117:0/3984485137 reap_dead start 2024-05-15T12:43:11.346 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.353+0000 7f8002ffd640 1 -- 172.21.15.117:0/3984485137 shutdown_connections 2024-05-15T12:43:11.346 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.353+0000 7f8002ffd640 1 -- 172.21.15.117:0/3984485137 >> 172.21.15.117:0/3984485137 conn(0x7f8010036620 msgr2=0x7f8010038590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:11.346 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.353+0000 7f8002ffd640 1 -- 172.21.15.117:0/3984485137 shutdown_connections 2024-05-15T12:43:11.346 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.353+0000 7f8002ffd640 1 -- 172.21.15.117:0/3984485137 wait complete. 2024-05-15T12:43:11.346 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.353+0000 7f8002ffd640 1 librados: shutdown 2024-05-15T12:43:11.347 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.355+0000 7f9ef359e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:11.347 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.355+0000 7f9ef359e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:11.347 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.355+0000 7f9ef359e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:43:11.347 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.355+0000 7f9ef359e640 1 -- 172.21.15.117:0/723854573 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9eec0057e0 con 0x7f9eec033930 2024-05-15T12:43:11.354 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.364+0000 7fedfc8ba640 10 monclient: get_monmap_and_config 2024-05-15T12:43:11.354 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.364+0000 7fedfc8ba640 10 monclient: build_initial_monmap 2024-05-15T12:43:11.354 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.364+0000 7fedfc8ba640 10 monclient: monmap: 2024-05-15T12:43:11.354 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:11.354 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:11.354 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:43:11.365341+0000 2024-05-15T12:43:11.354 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:43:11.365341+0000 2024-05-15T12:43:11.355 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:11.355 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.355 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T12:43:11.355 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T12:43:11.355 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T12:43:11.355 INFO:teuthology.orchestra.run.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T12:43:11.355 INFO:teuthology.orchestra.run.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T12:43:11.355 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.356 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.365+0000 7fedfc8ba640 1 Processor -- start 2024-05-15T12:43:11.356 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.365+0000 7fedfc8ba640 1 -- start start 2024-05-15T12:43:11.356 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.365+0000 7fedfc8ba640 10 monclient: init 2024-05-15T12:43:11.356 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.366+0000 7fedfc8ba640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:11.356 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.366+0000 7fedfc8ba640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-05-15T12:43:11.356 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.366+0000 7fedfc8ba640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fedf0006fd0 0x7fedf00cb150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.356 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.366+0000 7fedfc8ba640 10 monclient(hunting): picked mon.noname-a con 0x7fedf0006fd0 addr v2:172.21.15.117:3300/0 2024-05-15T12:43:11.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.366+0000 7fedfc8ba640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fedf00073b0 0x7fedf0035f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.366+0000 7fedfc8ba640 10 monclient(hunting): picked mon.noname-b con 0x7fedf00073b0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:43:11.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.366+0000 7fedfc8ba640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fedf0033930 addr v1:172.21.15.178:6789/0 2024-05-15T12:43:11.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.366+0000 7fedfc8ba640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedf0037050 con 0x7fedf0033930 2024-05-15T12:43:11.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.366+0000 7fedfc8ba640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:11.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.366+0000 7fedfc8ba640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedf00357e0 con 0x7fedf0006fd0 2024-05-15T12:43:11.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.366+0000 7fedfc8ba640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:11.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.366+0000 7fedfc8ba640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedf0035ac0 con 0x7fedf00073b0 2024-05-15T12:43:11.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.366+0000 7fedfc8ba640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:11.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.366+0000 7fedfc8ba640 10 monclient(hunting): authenticate will time out at 1354.608616s 2024-05-15T12:43:11.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.367+0000 7f4e505d1640 10 monclient: get_monmap_and_config 2024-05-15T12:43:11.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.367+0000 7f4e505d1640 10 monclient: build_initial_monmap 2024-05-15T12:43:11.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.367+0000 7f4e505d1640 10 monclient: monmap: 2024-05-15T12:43:11.358 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:11.358 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:11.358 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:43:11.368146+0000 2024-05-15T12:43:11.358 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:43:11.368146+0000 2024-05-15T12:43:11.358 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:11.358 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.358 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T12:43:11.358 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T12:43:11.359 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T12:43:11.359 INFO:teuthology.orchestra.run.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T12:43:11.359 INFO:teuthology.orchestra.run.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T12:43:11.359 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.367+0000 7fedf77fe640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fedf0006fd0 0x7fedf00cb150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:11.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.368+0000 7fedf6ffd640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fedf00073b0 0x7fedf0035f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:11.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.368+0000 7fedf6ffd640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fedf00073b0 0x7fedf0035f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.117:48954/0 (socket says 172.21.15.117:48954) 2024-05-15T12:43:11.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.368+0000 7fedf6ffd640 1 -- 172.21.15.117:0/91610635 learned_addr learned my addr 172.21.15.117:0/91610635 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:43:11.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.368+0000 7fedf6ffd640 10 monclient(hunting): get_auth_request con 0x7fedf00073b0 auth_method 0 2024-05-15T12:43:11.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.368+0000 7fedf6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:11.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.368+0000 7fedf6ffd640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:11.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.368+0000 7fedf6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:11.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.368+0000 7fd9e3a8f640 10 monclient: get_monmap_and_config 2024-05-15T12:43:11.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.368+0000 7fd9e3a8f640 10 monclient: build_initial_monmap 2024-05-15T12:43:11.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.368+0000 7fd9e3a8f640 10 monclient: monmap: 2024-05-15T12:43:11.360 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:11.360 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:11.360 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:43:11.369948+0000 2024-05-15T12:43:11.360 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:43:11.369948+0000 2024-05-15T12:43:11.361 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:11.361 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.361 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T12:43:11.361 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T12:43:11.361 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T12:43:11.361 INFO:teuthology.orchestra.run.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T12:43:11.361 INFO:teuthology.orchestra.run.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T12:43:11.361 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.369+0000 7fedf77fe640 10 monclient(hunting): get_auth_request con 0x7fedf0006fd0 auth_method 0 2024-05-15T12:43:11.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.369+0000 7fedf77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:11.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.369+0000 7fedf77fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:11.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.369+0000 7fedf77fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:11.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.368+0000 7f4e505d1640 1 Processor -- start 2024-05-15T12:43:11.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.369+0000 7fedf6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:11.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.368+0000 7f4e505d1640 1 -- start start 2024-05-15T12:43:11.363 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.369+0000 7fedf6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:11.363 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.370+0000 7fedf6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:11.363 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.370+0000 7fedf77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:11.363 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.370+0000 7fedf77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:11.363 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.370+0000 7fedf77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:11.364 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.368+0000 7f4e505d1640 10 monclient: init 2024-05-15T12:43:11.364 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.369+0000 7f4e505d1640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:11.364 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.369+0000 7f4e505d1640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-05-15T12:43:11.364 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.369+0000 7f4e505d1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4e480b1f30 addr v1:172.21.15.178:6789/0 2024-05-15T12:43:11.364 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.369+0000 7f4e505d1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4e48144500 addr v1:172.21.15.117:6789/0 2024-05-15T12:43:11.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.369+0000 7f4e505d1640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f4e480b3000 0x7f4e481523b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.369+0000 7f4e505d1640 10 monclient(hunting): picked mon.noname-a con 0x7f4e480b3000 addr v2:172.21.15.117:3300/0 2024-05-15T12:43:11.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.369+0000 7f4e505d1640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e4806f4d0 con 0x7f4e48144500 2024-05-15T12:43:11.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.369+0000 7f4e505d1640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e4806f7b0 con 0x7f4e480b1f30 2024-05-15T12:43:11.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.369+0000 7f4e505d1640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:11.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.369+0000 7f4e505d1640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e480b2310 con 0x7f4e480b3000 2024-05-15T12:43:11.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.369+0000 7f4e505d1640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:11.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.369+0000 7f4e505d1640 10 monclient(hunting): authenticate will time out at 1354.611354s 2024-05-15T12:43:11.366 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.370+0000 7fd9e3a8f640 1 Processor -- start 2024-05-15T12:43:11.366 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.370+0000 7fd9e3a8f640 1 -- start start 2024-05-15T12:43:11.366 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.370+0000 7fd9e3a8f640 10 monclient: init 2024-05-15T12:43:11.366 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.371+0000 7fd9e3a8f640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:11.366 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.371+0000 7fd9e3a8f640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-05-15T12:43:11.367 INFO:teuthology.orchestra.run.smithi117.stdout:64424509447 2024-05-15T12:43:11.367 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-05-15T12:43:11.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.371+0000 7fd9e3a8f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd9dc06bd70 addr v1:172.21.15.117:6789/0 2024-05-15T12:43:11.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.371+0000 7fd9e3a8f640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fd9dc06eac0 0x7fd9dc14a630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.371+0000 7fd9e3a8f640 10 monclient(hunting): picked mon.noname-c con 0x7fd9dc06eac0 addr v2:172.21.15.178:3300/0 2024-05-15T12:43:11.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.371+0000 7fd9e3a8f640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fd9dc14ab70 0x7fd9dc1514c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.371+0000 7fd9e3a8f640 10 monclient(hunting): picked mon.noname-a con 0x7fd9dc14ab70 addr v2:172.21.15.117:3300/0 2024-05-15T12:43:11.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.371+0000 7fd9e3a8f640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9dc070860 con 0x7fd9dc06bd70 2024-05-15T12:43:11.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.371+0000 7fd9e3a8f640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:11.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.371+0000 7fd9e3a8f640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9dc06c150 con 0x7fd9dc14ab70 2024-05-15T12:43:11.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.371+0000 7fd9e3a8f640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:11.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.371+0000 7fd9e3a8f640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9dc070ad0 con 0x7fd9dc06eac0 2024-05-15T12:43:11.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.371+0000 7fd9e3a8f640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:11.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.371+0000 7fd9e3a8f640 10 monclient(hunting): authenticate will time out at 1354.613649s 2024-05-15T12:43:11.369 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.372+0000 7fedf77fe640 10 monclient(hunting): handle_auth_done global_id 4308 payload 274 2024-05-15T12:43:11.369 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.372+0000 7fedf77fe640 1 -- 172.21.15.117:0/91610635 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fedf00073b0 msgr2=0x7fedf0035f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:43:11.369 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.372+0000 7fedf77fe640 1 --2- 172.21.15.117:0/91610635 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fedf00073b0 0x7fedf0035f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.369 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.372+0000 7fedf77fe640 1 -- 172.21.15.117:0/91610635 >> v1:172.21.15.178:6789/0 conn(0x7fedf0033930 legacy=0x7fedf0038db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:43:11.369 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.372+0000 7fedf77fe640 10 monclient: _finish_hunting 0 2024-05-15T12:43:11.369 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.372+0000 7fedf77fe640 1 monclient: found mon.noname-a 2024-05-15T12:43:11.369 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.372+0000 7fedf77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:11.369 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.372+0000 7fedf77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.117:3300/0 2024-05-15T12:43:11.369 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.372+0000 7fedf77fe640 1 -- 172.21.15.117:0/91610635 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedf0037330 con 0x7fedf0006fd0 2024-05-15T12:43:11.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.375+0000 7f4e4eb6e640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f4e480b3000 0x7f4e481523b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:11.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.375+0000 7f4e4eb6e640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f4e480b3000 0x7f4e481523b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:60184/0 (socket says 172.21.15.117:60184) 2024-05-15T12:43:11.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.375+0000 7f4e4eb6e640 1 -- 172.21.15.117:0/2002364893 learned_addr learned my addr 172.21.15.117:0/2002364893 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:43:11.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.375+0000 7f4e4eb6e640 10 monclient(hunting): get_auth_request con 0x7f4e480b3000 auth_method 0 2024-05-15T12:43:11.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.375+0000 7f4e4eb6e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:11.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.375+0000 7f4e4eb6e640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:11.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.375+0000 7f4e4eb6e640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:11.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.376+0000 7fedf77fe640 1 --2- 172.21.15.117:0/91610635 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fedf0006fd0 0x7fedf00cb150 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fedec009790 tx=0x7fedec02b780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b46bb4c3c4a552ea server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:11.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.376+0000 7f4e4eb6e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:11.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.376+0000 7f4e4eb6e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:11.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.376+0000 7f4e4eb6e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:11.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.378+0000 7fb482928640 10 monclient: get_monmap_and_config 2024-05-15T12:43:11.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.378+0000 7fb482928640 10 monclient: build_initial_monmap 2024-05-15T12:43:11.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.378+0000 7fb482928640 10 monclient: monmap: 2024-05-15T12:43:11.372 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:11.372 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:11.372 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:43:11.379167+0000 2024-05-15T12:43:11.372 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:43:11.379167+0000 2024-05-15T12:43:11.372 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:11.372 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.372 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T12:43:11.373 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T12:43:11.373 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T12:43:11.373 INFO:teuthology.orchestra.run.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T12:43:11.373 INFO:teuthology.orchestra.run.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T12:43:11.373 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.373 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.378+0000 7fb482928640 1 Processor -- start 2024-05-15T12:43:11.373 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.379+0000 7fb482928640 1 -- start start 2024-05-15T12:43:11.373 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.379+0000 7fb482928640 10 monclient: init 2024-05-15T12:43:11.373 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.379+0000 7f4e4eb6e640 10 monclient(hunting): handle_auth_done global_id 4311 payload 274 2024-05-15T12:43:11.373 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.379+0000 7f4e4eb6e640 1 -- 172.21.15.117:0/2002364893 >> v1:172.21.15.178:6789/0 conn(0x7f4e480b1f30 legacy=0x7f4e48143df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:43:11.373 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.379+0000 7f4e4eb6e640 1 -- 172.21.15.117:0/2002364893 >> v1:172.21.15.117:6789/0 conn(0x7f4e48144500 legacy=0x7f4e480b28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:43:11.373 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.379+0000 7f4e4eb6e640 10 monclient: _finish_hunting 0 2024-05-15T12:43:11.374 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.379+0000 7f4e4eb6e640 1 monclient: found mon.noname-a 2024-05-15T12:43:11.374 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.379+0000 7f4e4eb6e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:11.374 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.379+0000 7f4e4eb6e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.117:3300/0 2024-05-15T12:43:11.374 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.379+0000 7f4e4eb6e640 1 -- 172.21.15.117:0/2002364893 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e481473b0 con 0x7f4e480b3000 2024-05-15T12:43:11.375 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.379+0000 7fb482928640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:11.375 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.379+0000 7fb482928640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-05-15T12:43:11.375 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.379+0000 7fb482928640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb47c0b4d00 addr v1:172.21.15.117:6789/0 2024-05-15T12:43:11.375 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.379+0000 7fb482928640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fb47c0b50e0 0x7fb47c0b1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.375 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.379+0000 7fb482928640 10 monclient(hunting): picked mon.noname-c con 0x7fb47c0b50e0 addr v2:172.21.15.178:3300/0 2024-05-15T12:43:11.375 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.379+0000 7fb482928640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb47c0a94b0 0x7fb47c0b0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.375 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.379+0000 7fb482928640 10 monclient(hunting): picked mon.noname-b con 0x7fb47c0a94b0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:43:11.375 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.379+0000 7fb482928640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb47c0ab1b0 con 0x7fb47c0b4d00 2024-05-15T12:43:11.375 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.379+0000 7fb482928640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:11.375 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.379+0000 7fb482928640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb47c0b23c0 con 0x7fb47c0b50e0 2024-05-15T12:43:11.375 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.379+0000 7fb482928640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:11.375 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.379+0000 7fb482928640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c0b1520 con 0x7fb47c0a94b0 2024-05-15T12:43:11.375 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.379+0000 7fb482928640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:11.375 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.379+0000 7fb482928640 10 monclient(hunting): authenticate will time out at 1354.621666s 2024-05-15T12:43:11.376 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.380+0000 7fb480ec5640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb47c0a94b0 0x7fb47c0b0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:11.376 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.380+0000 7fd9e182b640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fd9dc06bd70 0x7fd9dc0711d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:48682/0 (socket says 172.21.15.117:48682) 2024-05-15T12:43:11.376 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.380+0000 7fd9e182b640 1 -- 172.21.15.117:0/83817810 learned_addr learned my addr 172.21.15.117:0/83817810 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-05-15T12:43:11.376 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.381+0000 7fb480ec5640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb47c0a94b0 0x7fb47c0b0fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.117:48962/0 (socket says 172.21.15.117:48962) 2024-05-15T12:43:11.377 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.381+0000 7fb480ec5640 1 -- 172.21.15.117:0/1752308507 learned_addr learned my addr 172.21.15.117:0/1752308507 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:43:11.377 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.381+0000 7fb480ec5640 10 monclient(hunting): get_auth_request con 0x7fb47c0a94b0 auth_method 0 2024-05-15T12:43:11.377 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.381+0000 7fb480ec5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:11.377 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.381+0000 7fb480ec5640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:11.377 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.381+0000 7fb480ec5640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:11.377 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.381+0000 7fb480ec5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:11.377 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.381+0000 7fb480ec5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:11.377 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.381+0000 7fb480ec5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:11.379 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.384+0000 7fb480ec5640 10 monclient(hunting): handle_auth_done global_id 4295 payload 274 2024-05-15T12:43:11.379 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.384+0000 7fb480ec5640 1 -- 172.21.15.117:0/1752308507 >> v2:172.21.15.178:3300/0 conn(0x7fb47c0b50e0 msgr2=0x7fb47c0b1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:43:11.379 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.384+0000 7fb480ec5640 1 --2- 172.21.15.117:0/1752308507 >> v2:172.21.15.178:3300/0 conn(0x7fb47c0b50e0 0x7fb47c0b1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.379 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.384+0000 7fb480ec5640 1 -- 172.21.15.117:0/1752308507 >> v1:172.21.15.117:6789/0 conn(0x7fb47c0b4d00 legacy=0x7fb47c145c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:43:11.379 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.384+0000 7fb480ec5640 10 monclient: _finish_hunting 0 2024-05-15T12:43:11.379 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.384+0000 7fb480ec5640 1 monclient: found mon.noname-b 2024-05-15T12:43:11.379 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.384+0000 7fb480ec5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:11.379 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.384+0000 7fb480ec5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.117:3301/0 2024-05-15T12:43:11.380 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.384+0000 7fb480ec5640 1 -- 172.21.15.117:0/1752308507 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb47c0b16d0 con 0x7fb47c0a94b0 2024-05-15T12:43:11.380 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.385+0000 7f4e4eb6e640 1 --2- 172.21.15.117:0/2002364893 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e480b3000 0x7f4e481523b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4e40009e50 tx=0x7f4e4002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94c953d6081e778 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:11.380 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.387+0000 7fd9e202c640 1 --2- 172.21.15.117:0/83817810 >> v2:172.21.15.117:3300/0 conn(0x7fd9dc14ab70 0x7fd9dc1514c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:11.380 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.388+0000 7fb480ec5640 1 --2- 172.21.15.117:0/1752308507 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb47c0a94b0 0x7fb47c0b0fe0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb474009e50 tx=0x7fb47402be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb82225f2b38502b server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:11.383 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.392+0000 7fb47affd640 1 -- 172.21.15.117:0/1752308507 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb47402c9e0 con 0x7fb47c0a94b0 2024-05-15T12:43:11.383 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.392+0000 7fb47affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:11.383 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.392+0000 7fb47affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:43:11.383 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.392+0000 7fb47affd640 10 monclient: dump: 2024-05-15T12:43:11.383 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:11.383 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:11.384 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.384 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.384 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:11.384 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.384 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:11.384 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:11.384 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:11.384 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.384 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.392+0000 7fb47affd640 10 monclient: _finish_auth 0 2024-05-15T12:43:11.384 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.392+0000 7fb47affd640 10 monclient: _check_auth_tickets 2024-05-15T12:43:11.384 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.392+0000 7fb47affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:11.384 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.392+0000 7fb482928640 5 monclient: authenticate success, global_id 4295 2024-05-15T12:43:11.385 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.392+0000 7fb482928640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:43:11.385 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.393+0000 7fb47affd640 1 -- 172.21.15.117:0/1752308507 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb47402cb80 con 0x7fb47c0a94b0 2024-05-15T12:43:11.385 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.393+0000 7fb47affd640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:11.385 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.393+0000 7fb47affd640 1 -- 172.21.15.117:0/1752308507 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4740343f0 con 0x7fb47c0a94b0 2024-05-15T12:43:11.385 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.393+0000 7fb482928640 10 monclient: get_monmap_and_config success 2024-05-15T12:43:11.385 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.393+0000 7fb482928640 4 set_mon_vals no callback set 2024-05-15T12:43:11.385 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.393+0000 7fd9e202c640 10 monclient(hunting): get_auth_request con 0x7fd9dc14ab70 auth_method 0 2024-05-15T12:43:11.385 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.393+0000 7fd9e202c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:11.385 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.393+0000 7fd9e202c640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:11.386 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.393+0000 7fd9e202c640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:11.386 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.394+0000 7fb47affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:11.386 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.394+0000 7fb47affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T12:43:11.386 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.394+0000 7fb47affd640 10 monclient: dump: 2024-05-15T12:43:11.386 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:11.386 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:11.386 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.386 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.386 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:11.387 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.387 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:11.387 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:11.387 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:11.387 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.387 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.394+0000 7fd9e202c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:11.387 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.394+0000 7fd9e202c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:11.388 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.394+0000 7fb482928640 10 monclient: shutdown 2024-05-15T12:43:11.388 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.394+0000 7fb482928640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:11.388 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.394+0000 7fb482928640 1 -- 172.21.15.117:0/1752308507 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb47c0a94b0 msgr2=0x7fb47c0b0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.388 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.394+0000 7fb482928640 1 --2- 172.21.15.117:0/1752308507 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb47c0a94b0 0x7fb47c0b0fe0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb474009e50 tx=0x7fb47402be40 comp rx=0 tx=0).stop 2024-05-15T12:43:11.388 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.394+0000 7fb482928640 1 -- 172.21.15.117:0/1752308507 shutdown_connections 2024-05-15T12:43:11.388 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.394+0000 7fb482928640 1 --2- 172.21.15.117:0/1752308507 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb47c0a94b0 0x7fb47c0b0fe0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.388 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.394+0000 7fb482928640 1 --2- 172.21.15.117:0/1752308507 >> v2:172.21.15.178:3300/0 conn(0x7fb47c0b50e0 0x7fb47c0b1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.388 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.394+0000 7fb482928640 1 -- 172.21.15.117:0/1752308507 >> 172.21.15.117:0/1752308507 conn(0x7fb47c066cc0 msgr2=0x7fb47c0670d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:11.388 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.394+0000 7fd9e202c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:11.389 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.394+0000 7fb482928640 1 -- 172.21.15.117:0/1752308507 shutdown_connections 2024-05-15T12:43:11.389 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.394+0000 7fb482928640 1 -- 172.21.15.117:0/1752308507 wait complete. 2024-05-15T12:43:11.389 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.394+0000 7fedf67fc640 1 -- 172.21.15.117:0/91610635 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedec037070 con 0x7fedf0006fd0 2024-05-15T12:43:11.389 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.394+0000 7fedf67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:11.389 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.394+0000 7fedf67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T12:43:11.389 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.394+0000 7fedf67fc640 10 monclient: dump: 2024-05-15T12:43:11.389 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:11.389 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:11.390 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.390 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.390 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:11.390 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.390 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:11.390 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:11.390 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:11.390 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.390 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.394+0000 7fedf67fc640 10 monclient: _finish_auth 0 2024-05-15T12:43:11.390 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.394+0000 7fedf67fc640 10 monclient: _check_auth_tickets 2024-05-15T12:43:11.390 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.394+0000 7fedf67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:11.390 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.394+0000 7fedf67fc640 1 -- 172.21.15.117:0/91610635 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedec030710 con 0x7fedf0006fd0 2024-05-15T12:43:11.390 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.394+0000 7fedf67fc640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:11.390 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.394+0000 7fedf67fc640 1 -- 172.21.15.117:0/91610635 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedec030ae0 con 0x7fedf0006fd0 2024-05-15T12:43:11.391 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.395+0000 7fedf67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:11.391 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.395+0000 7fedf67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T12:43:11.391 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.395+0000 7fedf67fc640 10 monclient: dump: 2024-05-15T12:43:11.391 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:11.391 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:11.391 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.391 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.391 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:11.391 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.391 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:11.391 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:11.391 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:11.391 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.392 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.395+0000 7fd9e0829640 1 -- 172.21.15.117:0/83817810 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 113578076 0 0) 0x7fd9dc070860 con 0x7fd9dc06bd70 2024-05-15T12:43:11.392 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.395+0000 7fd9e0829640 10 monclient(hunting): _negotiate 2024-05-15T12:43:11.392 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.395+0000 7fd9e0829640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:11.392 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.395+0000 7fd9e0829640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:11.392 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.395+0000 7fd9e0829640 10 monclient(hunting): my global_id is 4314 2024-05-15T12:43:11.392 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.395+0000 7fd9e0829640 1 -- 172.21.15.117:0/83817810 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9c8003520 con 0x7fd9dc06bd70 2024-05-15T12:43:11.392 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.395+0000 7fb482928640 10 monclient: build_initial_monmap 2024-05-15T12:43:11.392 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.395+0000 7fb482928640 10 monclient: monmap: 2024-05-15T12:43:11.392 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:11.392 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:43:11.392 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-05-15T12:43:11.393 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-05-15T12:43:11.393 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:11.393 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.393 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T12:43:11.393 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T12:43:11.393 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T12:43:11.393 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.393 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.395+0000 7fedfc8ba640 5 monclient: authenticate success, global_id 4308 2024-05-15T12:43:11.394 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.395+0000 7fedfc8ba640 10 monclient: get_monmap_and_config success 2024-05-15T12:43:11.394 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.395+0000 7fedfc8ba640 4 set_mon_vals no callback set 2024-05-15T12:43:11.394 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.395+0000 7fedfc8ba640 10 monclient: shutdown 2024-05-15T12:43:11.394 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.395+0000 7fedfc8ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:11.394 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.395+0000 7fedfc8ba640 1 -- 172.21.15.117:0/91610635 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fedf0006fd0 msgr2=0x7fedf00cb150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.394 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.395+0000 7fedfc8ba640 1 --2- 172.21.15.117:0/91610635 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fedf0006fd0 0x7fedf00cb150 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fedec009790 tx=0x7fedec02b780 comp rx=0 tx=0).stop 2024-05-15T12:43:11.394 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.396+0000 7fedfc8ba640 1 -- 172.21.15.117:0/91610635 shutdown_connections 2024-05-15T12:43:11.394 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.396+0000 7fedfc8ba640 1 --2- 172.21.15.117:0/91610635 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fedf00073b0 0x7fedf0035f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.394 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.396+0000 7fedfc8ba640 1 --2- 172.21.15.117:0/91610635 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fedf0006fd0 0x7fedf00cb150 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.394 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.396+0000 7fedfc8ba640 1 -- 172.21.15.117:0/91610635 >> 172.21.15.117:0/91610635 conn(0x7fedf00061a0 msgr2=0x7fedf00065b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:11.394 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.396+0000 7fedfc8ba640 1 -- 172.21.15.117:0/91610635 shutdown_connections 2024-05-15T12:43:11.395 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.396+0000 7fb482928640 1 librados: starting msgr at 2024-05-15T12:43:11.395 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.396+0000 7fb482928640 1 librados: starting objecter 2024-05-15T12:43:11.395 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.396+0000 7fb482928640 1 Processor -- start 2024-05-15T12:43:11.395 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.396+0000 7fb482928640 1 -- start start 2024-05-15T12:43:11.395 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.396+0000 7fedfc8ba640 1 -- 172.21.15.117:0/91610635 wait complete. 2024-05-15T12:43:11.396 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.396+0000 7fb482928640 1 librados: setting wanted keys 2024-05-15T12:43:11.396 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.396+0000 7fb482928640 1 librados: calling monclient init 2024-05-15T12:43:11.396 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.397+0000 7fb482928640 10 monclient: init 2024-05-15T12:43:11.396 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.397+0000 7fedfc8ba640 10 monclient: build_initial_monmap 2024-05-15T12:43:11.396 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.397+0000 7fedfc8ba640 10 monclient: monmap: 2024-05-15T12:43:11.397 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:11.397 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:43:11.397 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-05-15T12:43:11.397 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-05-15T12:43:11.397 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:11.397 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.397 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T12:43:11.397 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T12:43:11.397 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T12:43:11.397 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.397+0000 7fedfc8ba640 1 librados: starting msgr at 2024-05-15T12:43:11.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.397+0000 7fedfc8ba640 1 librados: starting objecter 2024-05-15T12:43:11.398 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.397+0000 7fb482928640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:11.398 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.397+0000 7fedfc8ba640 1 Processor -- start 2024-05-15T12:43:11.398 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.397+0000 7fb482928640 10 monclient: _add_conns ranks=[2,0,1] 2024-05-15T12:43:11.399 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.397+0000 7fd9e202c640 10 monclient(hunting): handle_auth_done global_id 4317 payload 274 2024-05-15T12:43:11.399 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.397+0000 7fd9e202c640 1 -- 172.21.15.117:0/83817810 >> v2:172.21.15.178:3300/0 conn(0x7fd9dc06eac0 msgr2=0x7fd9dc14a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:43:11.399 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.397+0000 7fd9e202c640 1 --2- 172.21.15.117:0/83817810 >> v2:172.21.15.178:3300/0 conn(0x7fd9dc06eac0 0x7fd9dc14a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.400 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.397+0000 7fd9e202c640 1 -- 172.21.15.117:0/83817810 >> v1:172.21.15.117:6789/0 conn(0x7fd9dc06bd70 legacy=0x7fd9dc0711d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.400 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.398+0000 7fd9e202c640 10 monclient: _finish_hunting 0 2024-05-15T12:43:11.400 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.398+0000 7fd9e202c640 1 monclient: found mon.noname-a 2024-05-15T12:43:11.400 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.398+0000 7fd9e202c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:11.400 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.398+0000 7fd9e202c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.117:3300/0 2024-05-15T12:43:11.400 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.398+0000 7fd9e202c640 1 -- 172.21.15.117:0/83817810 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9dc0718e0 con 0x7fd9dc14ab70 2024-05-15T12:43:11.401 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.398+0000 7fedfc8ba640 1 -- start start 2024-05-15T12:43:11.401 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.398+0000 7fedfc8ba640 1 librados: setting wanted keys 2024-05-15T12:43:11.401 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.398+0000 7fedfc8ba640 1 librados: calling monclient init 2024-05-15T12:43:11.401 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.398+0000 7fedfc8ba640 10 monclient: init 2024-05-15T12:43:11.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.397+0000 7fb482928640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb47c0a94b0 0x7fb47c1418d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.398+0000 7fb482928640 10 monclient(hunting): picked mon.noname-c con 0x7fb47c0a94b0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:43:11.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.398+0000 7fb482928640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb47c0b4d00 0x7fb47c1d8c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.398+0000 7fb482928640 10 monclient(hunting): picked mon.noname-a con 0x7fb47c0b4d00 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T12:43:11.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.398+0000 7fb482928640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb47c13a1a0 0x7fb47c13a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.398+0000 7fb482928640 10 monclient(hunting): picked mon.noname-b con 0x7fb47c13a1a0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:43:11.403 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.399+0000 7fedfc8ba640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:11.403 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.399+0000 7fedfc8ba640 10 monclient: _add_conns ranks=[0,1,2] 2024-05-15T12:43:11.403 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.399+0000 7fedfc8ba640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fedf0006fd0 0x7fedf0154710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.403 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.399+0000 7fb482928640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:11.404 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.399+0000 7fb482928640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c14d130 con 0x7fb47c0b4d00 2024-05-15T12:43:11.404 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.399+0000 7fb482928640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:11.404 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.399+0000 7fb482928640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c14cfb0 con 0x7fb47c13a1a0 2024-05-15T12:43:11.404 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.399+0000 7fb482928640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:11.404 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.399+0000 7fb482928640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c14d2b0 con 0x7fb47c0a94b0 2024-05-15T12:43:11.404 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.399+0000 7fb482928640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:11.404 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.399+0000 7fb482928640 10 monclient(hunting): authenticate will time out at 1354.641118s 2024-05-15T12:43:11.404 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.399+0000 7fedfc8ba640 10 monclient(hunting): picked mon.noname-a con 0x7fedf0006fd0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T12:43:11.404 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.399+0000 7fedfc8ba640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fedf0033930 0x7fedf0151b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.405 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.399+0000 7fb47bfff640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb47c0a94b0 0x7fb47c1418d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:11.405 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.399+0000 7fedfc8ba640 10 monclient(hunting): picked mon.noname-b con 0x7fedf0033930 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:43:11.406 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.399+0000 7fedfc8ba640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fedf00c64a0 0x7fedf0152070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.406 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.399+0000 7fedfc8ba640 10 monclient(hunting): picked mon.noname-c con 0x7fedf00c64a0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:43:11.406 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.399+0000 7fedfc8ba640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:11.406 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.399+0000 7fedfc8ba640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf0036610 con 0x7fedf0006fd0 2024-05-15T12:43:11.406 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.399+0000 7fedfc8ba640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:11.406 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.399+0000 7fedfc8ba640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf0036490 con 0x7fedf0033930 2024-05-15T12:43:11.406 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.399+0000 7fedfc8ba640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:11.406 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.399+0000 7fedfc8ba640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedf0030bf0 con 0x7fedf00c64a0 2024-05-15T12:43:11.406 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.399+0000 7fedfc8ba640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:11.406 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.399+0000 7fedfc8ba640 10 monclient(hunting): authenticate will time out at 1354.641170s 2024-05-15T12:43:11.407 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.399+0000 7fb47bfff640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb47c0a94b0 0x7fb47c1418d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.117:48966/0 (socket says 172.21.15.117:48966) 2024-05-15T12:43:11.407 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.399+0000 7fb47bfff640 1 -- 172.21.15.117:0/1085814967 learned_addr learned my addr 172.21.15.117:0/1085814967 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:43:11.407 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.399+0000 7fb47bfff640 10 monclient(hunting): get_auth_request con 0x7fb47c0a94b0 auth_method 0 2024-05-15T12:43:11.407 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.399+0000 7fb47bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:11.407 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.399+0000 7fb47bfff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:11.407 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.399+0000 7fb47bfff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:11.407 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.400+0000 7fedf7fff640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fedf00c64a0 0x7fedf0152070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:11.408 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.400+0000 7fedf7fff640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fedf00c64a0 0x7fedf0152070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.117:48982/0 (socket says 172.21.15.117:48982) 2024-05-15T12:43:11.408 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.400+0000 7fedf7fff640 1 -- 172.21.15.117:0/1283422087 learned_addr learned my addr 172.21.15.117:0/1283422087 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:43:11.408 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.400+0000 7fedf7fff640 10 monclient(hunting): get_auth_request con 0x7fedf00c64a0 auth_method 0 2024-05-15T12:43:11.408 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.400+0000 7fedf7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:11.408 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.400+0000 7fedf7fff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:11.408 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.400+0000 7fedf7fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:11.409 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.400+0000 7f4e4d36b640 1 -- 172.21.15.117:0/2002364893 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e4002c9f0 con 0x7f4e480b3000 2024-05-15T12:43:11.409 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.400+0000 7f4e4d36b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:11.409 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.400+0000 7f4e4d36b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T12:43:11.409 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.400+0000 7f4e4d36b640 10 monclient: dump: 2024-05-15T12:43:11.409 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:11.409 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:11.409 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.409 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.409 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:11.409 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.409 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:11.409 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:11.409 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:11.410 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.410 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.400+0000 7f4e4d36b640 10 monclient: _finish_auth 0 2024-05-15T12:43:11.410 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.400+0000 7f4e4d36b640 10 monclient: _check_auth_tickets 2024-05-15T12:43:11.410 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.400+0000 7f4e4d36b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:11.410 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.400+0000 7f4e4d36b640 1 -- 172.21.15.117:0/2002364893 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e4002cb90 con 0x7f4e480b3000 2024-05-15T12:43:11.410 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.400+0000 7f4e4d36b640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:11.410 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.400+0000 7f4e4d36b640 1 -- 172.21.15.117:0/2002364893 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e400343f0 con 0x7f4e480b3000 2024-05-15T12:43:11.410 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.400+0000 7f4e4d36b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:11.410 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.400+0000 7f4e4d36b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T12:43:11.410 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.400+0000 7f4e4d36b640 10 monclient: dump: 2024-05-15T12:43:11.410 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:11.410 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:11.410 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.410 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.410 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:11.411 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.411 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:11.411 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:11.411 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:11.411 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.411 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.400+0000 7f4e505d1640 5 monclient: authenticate success, global_id 4311 2024-05-15T12:43:11.411 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.400+0000 7f4e505d1640 10 monclient: get_monmap_and_config success 2024-05-15T12:43:11.411 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.400+0000 7f4e505d1640 4 set_mon_vals no callback set 2024-05-15T12:43:11.411 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.401+0000 7fedf7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:11.412 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.401+0000 7fedf7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:11.412 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.401+0000 7fedf7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:11.412 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.401+0000 7f4e505d1640 10 monclient: shutdown 2024-05-15T12:43:11.412 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.401+0000 7f4e505d1640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:11.412 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.401+0000 7f4e505d1640 1 -- 172.21.15.117:0/2002364893 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e480b3000 msgr2=0x7f4e481523b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.412 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.401+0000 7f4e505d1640 1 --2- 172.21.15.117:0/2002364893 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e480b3000 0x7f4e481523b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4e40009e50 tx=0x7f4e4002be50 comp rx=0 tx=0).stop 2024-05-15T12:43:11.412 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.401+0000 7f4e505d1640 1 -- 172.21.15.117:0/2002364893 shutdown_connections 2024-05-15T12:43:11.412 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.401+0000 7f4e505d1640 1 --2- 172.21.15.117:0/2002364893 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e480b3000 0x7f4e481523b0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.413 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.401+0000 7f4e505d1640 1 -- 172.21.15.117:0/2002364893 >> 172.21.15.117:0/2002364893 conn(0x7f4e48066cc0 msgr2=0x7f4e480670d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:11.413 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.401+0000 7fedf77fe640 1 --2- 172.21.15.117:0/1283422087 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fedf0006fd0 0x7fedf0154710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:11.415 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.402+0000 7f4e505d1640 1 -- 172.21.15.117:0/2002364893 shutdown_connections 2024-05-15T12:43:11.415 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.402+0000 7f4e505d1640 1 -- 172.21.15.117:0/2002364893 wait complete. 2024-05-15T12:43:11.415 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.402+0000 7f4e505d1640 10 monclient: build_initial_monmap 2024-05-15T12:43:11.415 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.402+0000 7f4e505d1640 10 monclient: monmap: 2024-05-15T12:43:11.415 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:11.415 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:43:11.415 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-05-15T12:43:11.416 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-05-15T12:43:11.416 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:11.416 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.416 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T12:43:11.416 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T12:43:11.416 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T12:43:11.416 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.416 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.403+0000 7f4e505d1640 1 librados: starting msgr at 2024-05-15T12:43:11.416 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.403+0000 7f4e505d1640 1 librados: starting objecter 2024-05-15T12:43:11.416 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.403+0000 7f4e505d1640 1 Processor -- start 2024-05-15T12:43:11.416 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.403+0000 7f4e505d1640 1 -- start start 2024-05-15T12:43:11.416 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.403+0000 7f4e505d1640 1 librados: setting wanted keys 2024-05-15T12:43:11.416 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.403+0000 7f4e505d1640 1 librados: calling monclient init 2024-05-15T12:43:11.416 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.403+0000 7f4e505d1640 10 monclient: init 2024-05-15T12:43:11.416 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.404+0000 7f4e505d1640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:11.416 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.404+0000 7f4e505d1640 10 monclient: _add_conns ranks=[2,0,1] 2024-05-15T12:43:11.417 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.404+0000 7f4e505d1640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4e480b1f30 0x7f4e480afa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.417 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.404+0000 7f4e505d1640 10 monclient(hunting): picked mon.noname-c con 0x7f4e480b1f30 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:43:11.417 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.404+0000 7f4e505d1640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e480b3000 0x7f4e481308e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.417 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.404+0000 7f4e505d1640 10 monclient(hunting): picked mon.noname-a con 0x7f4e480b3000 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T12:43:11.417 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.404+0000 7f4e505d1640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4e48144500 0x7f4e48130e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.417 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.404+0000 7f4e505d1640 10 monclient(hunting): picked mon.noname-b con 0x7f4e48144500 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:43:11.417 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.404+0000 7f4e505d1640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:11.417 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.404+0000 7f4e505d1640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e48147a50 con 0x7f4e480b3000 2024-05-15T12:43:11.417 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.404+0000 7f4e505d1640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:11.417 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.404+0000 7f4e505d1640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e481478d0 con 0x7f4e48144500 2024-05-15T12:43:11.417 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.404+0000 7f4e505d1640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:11.417 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.404+0000 7f4e505d1640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4806e4b0 con 0x7f4e480b1f30 2024-05-15T12:43:11.417 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.404+0000 7f4e505d1640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:11.417 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.404+0000 7f4e505d1640 10 monclient(hunting): authenticate will time out at 1354.646730s 2024-05-15T12:43:11.419 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.405+0000 7fedf77fe640 10 monclient(hunting): get_auth_request con 0x7fedf0006fd0 auth_method 0 2024-05-15T12:43:11.419 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.405+0000 7fedf77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:11.420 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.405+0000 7fedf77fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:11.420 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.405+0000 7fedf77fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:11.420 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.406+0000 7fedf77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:11.420 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.406+0000 7fedf77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:11.420 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.406+0000 7fedf77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:11.420 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.406+0000 7fedf7fff640 10 monclient(hunting): handle_auth_done global_id 4298 payload 824 2024-05-15T12:43:11.420 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.406+0000 7fedf7fff640 1 -- 172.21.15.117:0/1283422087 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fedf0033930 msgr2=0x7fedf0151b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.420 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.406+0000 7fedf7fff640 1 --2- 172.21.15.117:0/1283422087 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fedf0033930 0x7fedf0151b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.420 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.406+0000 7fedf7fff640 1 -- 172.21.15.117:0/1283422087 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fedf0006fd0 msgr2=0x7fedf0154710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.420 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.406+0000 7fedf7fff640 1 --2- 172.21.15.117:0/1283422087 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fedf0006fd0 0x7fedf0154710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.420 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.406+0000 7fedf7fff640 10 monclient: _finish_hunting 0 2024-05-15T12:43:11.420 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.406+0000 7fedf7fff640 1 monclient: found mon.noname-c 2024-05-15T12:43:11.420 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.406+0000 7fedf7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:11.420 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.406+0000 7fedf7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.117:3301/0 2024-05-15T12:43:11.420 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.406+0000 7fedf7fff640 1 -- 172.21.15.117:0/1283422087 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedf00c76f0 con 0x7fedf00c64a0 2024-05-15T12:43:11.421 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.409+0000 7fb47b7fe640 1 --2- 172.21.15.117:0/1085814967 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb47c0b4d00 0x7fb47c1d8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:11.421 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.410+0000 7fedf7fff640 1 --2- 172.21.15.117:0/1283422087 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fedf00c64a0 0x7fedf0152070 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fedf80508c0 tx=0x7fedf806a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:11.421 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.410+0000 7fede3fff640 10 client.?.objecter ms_handle_connect 0x7fedf00c64a0 2024-05-15T12:43:11.421 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.410+0000 7fede3fff640 10 client.?.objecter resend_mon_ops 2024-05-15T12:43:11.422 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.411+0000 7fb47bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:11.422 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.411+0000 7fb47bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:11.422 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.411+0000 7fb47bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:11.422 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.413+0000 7f19e6820640 10 monclient: get_monmap_and_config 2024-05-15T12:43:11.423 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.413+0000 7f19e6820640 10 monclient: build_initial_monmap 2024-05-15T12:43:11.423 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.413+0000 7f19e6820640 10 monclient: monmap: 2024-05-15T12:43:11.423 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:11.423 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:11.423 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:43:11.414359+0000 2024-05-15T12:43:11.423 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:43:11.414359+0000 2024-05-15T12:43:11.423 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:11.423 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.423 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T12:43:11.423 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T12:43:11.423 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T12:43:11.423 INFO:teuthology.orchestra.run.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T12:43:11.423 INFO:teuthology.orchestra.run.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T12:43:11.423 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.424 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.414+0000 7f19e6820640 1 Processor -- start 2024-05-15T12:43:11.424 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.414+0000 7f19e6820640 1 -- start start 2024-05-15T12:43:11.424 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.414+0000 7f19e6820640 10 monclient: init 2024-05-15T12:43:11.424 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.415+0000 7fb47bfff640 10 monclient(hunting): handle_auth_done global_id 4301 payload 824 2024-05-15T12:43:11.424 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.415+0000 7fb47bfff640 1 -- 172.21.15.117:0/1085814967 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb47c13a1a0 msgr2=0x7fb47c13a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.424 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.415+0000 7fb47bfff640 1 --2- 172.21.15.117:0/1085814967 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb47c13a1a0 0x7fb47c13a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.425 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.415+0000 7fb47bfff640 1 -- 172.21.15.117:0/1085814967 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb47c0b4d00 msgr2=0x7fb47c1d8c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.425 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.415+0000 7fb47bfff640 1 --2- 172.21.15.117:0/1085814967 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb47c0b4d00 0x7fb47c1d8c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.425 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.415+0000 7fb47bfff640 10 monclient: _finish_hunting 0 2024-05-15T12:43:11.425 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.415+0000 7fb47bfff640 1 monclient: found mon.noname-c 2024-05-15T12:43:11.425 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.415+0000 7fb47bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:11.425 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.415+0000 7fb47bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.117:3301/0 2024-05-15T12:43:11.425 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.415+0000 7fb47bfff640 1 -- 172.21.15.117:0/1085814967 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb47c1d8770 con 0x7fb47c0a94b0 2024-05-15T12:43:11.426 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.415+0000 7f19e6820640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:11.426 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.415+0000 7f19e6820640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-05-15T12:43:11.426 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.415+0000 7f19e6820640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f19d8035250 addr v1:172.21.15.178:6789/0 2024-05-15T12:43:11.426 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.415+0000 7f19e6820640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f19d8035630 addr v1:172.21.15.117:6789/0 2024-05-15T12:43:11.426 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.415+0000 7f19e6820640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f19d80c45c0 0x7f19d80c4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.426 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.415+0000 7f19e6820640 10 monclient(hunting): picked mon.noname-c con 0x7f19d80c45c0 addr v2:172.21.15.178:3300/0 2024-05-15T12:43:11.426 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.415+0000 7f19e6820640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19d802d480 con 0x7f19d8035630 2024-05-15T12:43:11.426 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.415+0000 7f19e6820640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19d80c76e0 con 0x7f19d8035250 2024-05-15T12:43:11.426 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.415+0000 7f19e6820640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:11.426 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.415+0000 7f19e6820640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19d80c1e10 con 0x7f19d80c45c0 2024-05-15T12:43:11.426 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.415+0000 7f19e6820640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:11.427 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.415+0000 7f19e6820640 10 monclient(hunting): authenticate will time out at 1354.657661s 2024-05-15T12:43:11.427 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.420+0000 7fb47bfff640 1 --2- 172.21.15.117:0/1085814967 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb47c0a94b0 0x7fb47c1418d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb46c00c410 tx=0x7fb46c00c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:11.427 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.420+0000 7fb478ff9640 10 client.?.objecter ms_handle_connect 0x7fb47c0a94b0 2024-05-15T12:43:11.427 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.420+0000 7fb478ff9640 10 client.?.objecter resend_mon_ops 2024-05-15T12:43:11.427 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.422+0000 7fd9e202c640 1 --2- 172.21.15.117:0/83817810 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9dc14ab70 0x7fd9dc1514c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc0adb20 tx=0x7fd9d802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1200562e6033090f server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:11.429 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.424+0000 7fede3fff640 1 -- 172.21.15.117:0/1283422087 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf80731c0 con 0x7fedf00c64a0 2024-05-15T12:43:11.429 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.424+0000 7fede3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:11.429 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.424+0000 7fede3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:43:11.429 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.424+0000 7fede3fff640 10 monclient: dump: 2024-05-15T12:43:11.429 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:11.430 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:11.430 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.430 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.430 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:11.430 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.430 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:11.430 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:11.430 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:11.430 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.430 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.424+0000 7fede3fff640 10 monclient: _finish_auth 0 2024-05-15T12:43:11.430 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.424+0000 7fede3fff640 10 monclient: _check_auth_tickets 2024-05-15T12:43:11.430 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.424+0000 7fede3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:11.430 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.424+0000 7fede3fff640 1 -- 172.21.15.117:0/1283422087 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedf8073360 con 0x7fedf00c64a0 2024-05-15T12:43:11.431 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.424+0000 7fedfc8ba640 5 monclient: authenticate success, global_id 4298 2024-05-15T12:43:11.431 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.424+0000 7fedfc8ba640 10 monclient: _renew_subs 2024-05-15T12:43:11.431 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.424+0000 7fedfc8ba640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T12:43:11.431 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.424+0000 7fedfc8ba640 1 -- 172.21.15.117:0/1283422087 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedf0154c50 con 0x7fedf00c64a0 2024-05-15T12:43:11.431 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.424+0000 7fede3fff640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:11.431 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.424+0000 7fede3fff640 1 -- 172.21.15.117:0/1283422087 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf8093400 con 0x7fedf00c64a0 2024-05-15T12:43:11.431 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.424+0000 7fede3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:11.431 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.424+0000 7fede3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T12:43:11.431 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.424+0000 7fede3fff640 10 monclient: dump: 2024-05-15T12:43:11.431 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:11.431 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:11.431 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.431 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.431 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:11.432 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.432 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:11.432 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:11.432 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:11.432 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.432 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.424+0000 7fedfc8ba640 10 client.4298.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:43:11.432 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.424+0000 7fedfc8ba640 10 monclient: _renew_subs 2024-05-15T12:43:11.432 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.424+0000 7fedfc8ba640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T12:43:11.432 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.424+0000 7fedfc8ba640 1 -- 172.21.15.117:0/1283422087 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedf00c5bb0 con 0x7fedf00c64a0 2024-05-15T12:43:11.432 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.424+0000 7fedfc8ba640 1 librados: init done 2024-05-15T12:43:11.432 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.424+0000 7fedf5ffb640 4 set_mon_vals no callback set 2024-05-15T12:43:11.432 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.425+0000 7f4e4e36d640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4e480b1f30 0x7f4e480afa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:11.433 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.425+0000 7fedfc8ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:11.433 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.425+0000 7fedfc8ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:11.433 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.425+0000 7fedfc8ba640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T12:43:11.433 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.425+0000 7fedfc8ba640 1 -- 172.21.15.117:0/1283422087 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedf00057e0 con 0x7fedf00c64a0 2024-05-15T12:43:11.434 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.425+0000 7f4e4e36d640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4e480b1f30 0x7f4e480afa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.117:48988/0 (socket says 172.21.15.117:48988) 2024-05-15T12:43:11.434 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.425+0000 7f4e4e36d640 1 -- 172.21.15.117:0/1274867206 learned_addr learned my addr 172.21.15.117:0/1274867206 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:43:11.434 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.425+0000 7f4e4e36d640 10 monclient(hunting): get_auth_request con 0x7f4e480b1f30 auth_method 0 2024-05-15T12:43:11.434 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.425+0000 7f4e4e36d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:11.434 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.425+0000 7f4e4e36d640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:11.434 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.425+0000 7f4e4e36d640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:11.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.426+0000 7fb478ff9640 1 -- 172.21.15.117:0/1085814967 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46c017070 con 0x7fb47c0a94b0 2024-05-15T12:43:11.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.426+0000 7fb478ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:11.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.426+0000 7fb478ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:43:11.436 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.426+0000 7fb478ff9640 10 monclient: dump: 2024-05-15T12:43:11.436 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:11.436 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:11.436 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.436 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.436 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:11.436 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.436 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:11.436 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:11.436 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:11.436 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.436 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.426+0000 7fb478ff9640 10 monclient: _finish_auth 0 2024-05-15T12:43:11.436 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.426+0000 7fb478ff9640 10 monclient: _check_auth_tickets 2024-05-15T12:43:11.436 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.426+0000 7fb478ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:11.436 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.426+0000 7fb478ff9640 1 -- 172.21.15.117:0/1085814967 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb46c0035f0 con 0x7fb47c0a94b0 2024-05-15T12:43:11.436 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.426+0000 7fb482928640 5 monclient: authenticate success, global_id 4301 2024-05-15T12:43:11.437 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.426+0000 7fb478ff9640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:11.437 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.426+0000 7fb478ff9640 1 -- 172.21.15.117:0/1085814967 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46c0039e0 con 0x7fb47c0a94b0 2024-05-15T12:43:11.437 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.426+0000 7fb478ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:11.437 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.426+0000 7fb478ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T12:43:11.437 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.426+0000 7fb478ff9640 10 monclient: dump: 2024-05-15T12:43:11.437 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:11.437 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:11.437 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.437 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.437 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:11.437 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.437 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:11.437 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:11.437 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:11.437 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.438 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.426+0000 7fb482928640 10 monclient: _renew_subs 2024-05-15T12:43:11.438 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.426+0000 7fb482928640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T12:43:11.438 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.426+0000 7fb482928640 1 -- 172.21.15.117:0/1085814967 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb47c13ab90 con 0x7fb47c0a94b0 2024-05-15T12:43:11.438 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.427+0000 7fb47a7fc640 4 set_mon_vals no callback set 2024-05-15T12:43:11.438 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.427+0000 7fb482928640 10 client.4301.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:43:11.438 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.427+0000 7fb482928640 10 monclient: _renew_subs 2024-05-15T12:43:11.438 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.427+0000 7fb482928640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T12:43:11.438 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.427+0000 7fb482928640 1 -- 172.21.15.117:0/1085814967 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb47c0ac860 con 0x7fb47c0a94b0 2024-05-15T12:43:11.438 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.427+0000 7fb482928640 1 librados: init done 2024-05-15T12:43:11.438 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.427+0000 7f4e4e36d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:11.438 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.427+0000 7f4e4e36d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:11.439 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.427+0000 7f4e4e36d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:11.439 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.428+0000 7fb482928640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:11.439 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.428+0000 7fb482928640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:11.439 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.428+0000 7fb482928640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T12:43:11.439 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.428+0000 7fb482928640 1 -- 172.21.15.117:0/1085814967 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb47c04f0e0 con 0x7fb47c0a94b0 2024-05-15T12:43:11.440 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.428+0000 7f4e4db6c640 1 --2- 172.21.15.117:0/1274867206 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e480b3000 0x7f4e481308e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:11.440 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.429+0000 7f4e4db6c640 10 monclient(hunting): get_auth_request con 0x7f4e480b3000 auth_method 0 2024-05-15T12:43:11.440 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.429+0000 7f4e4db6c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:11.440 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.429+0000 7f4e4db6c640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:11.440 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.429+0000 7f4e4db6c640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:11.440 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.429+0000 7f19e501d640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f19d8035630 0x7f19d8031e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:48700/0 (socket says 172.21.15.117:48700) 2024-05-15T12:43:11.440 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.429+0000 7f19e501d640 1 -- 172.21.15.117:0/1789255093 learned_addr learned my addr 172.21.15.117:0/1789255093 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-05-15T12:43:11.441 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.430+0000 7f4e4db6c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:11.441 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.430+0000 7f4e4db6c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:11.441 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.430+0000 7f4e4db6c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:11.442 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.431+0000 7fd9e0829640 1 -- 172.21.15.117:0/83817810 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d80315a0 con 0x7fd9dc14ab70 2024-05-15T12:43:11.442 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.431+0000 7fd9e0829640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:11.442 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.431+0000 7fd9e0829640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T12:43:11.442 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.431+0000 7fd9e0829640 10 monclient: dump: 2024-05-15T12:43:11.442 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:11.442 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:11.443 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.443 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.443 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:11.443 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.443 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:11.443 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:11.443 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:11.443 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.431+0000 7fd9e0829640 10 monclient: _finish_auth 0 2024-05-15T12:43:11.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.431+0000 7fd9e0829640 10 monclient: _check_auth_tickets 2024-05-15T12:43:11.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.431+0000 7fd9e0829640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:11.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.431+0000 7fd9e3a8f640 5 monclient: authenticate success, global_id 4317 2024-05-15T12:43:11.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.431+0000 7fd9e3a8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:43:11.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.431+0000 7fd9e0829640 1 -- 172.21.15.117:0/83817810 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9d8037070 con 0x7fd9dc14ab70 2024-05-15T12:43:11.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.431+0000 7fd9e0829640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:11.444 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.431+0000 7fd9e3a8f640 10 monclient: get_monmap_and_config success 2024-05-15T12:43:11.444 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.431+0000 7fd9e3a8f640 4 set_mon_vals no callback set 2024-05-15T12:43:11.444 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.431+0000 7fd9e0829640 1 -- 172.21.15.117:0/83817810 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d80315a0 con 0x7fd9dc14ab70 2024-05-15T12:43:11.444 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.431+0000 7fd9e0829640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:11.444 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.431+0000 7fd9e0829640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T12:43:11.444 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.431+0000 7fd9e0829640 10 monclient: dump: 2024-05-15T12:43:11.444 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:11.444 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:11.444 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.444 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.444 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:11.444 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.444 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:11.445 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:11.445 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:11.445 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.445 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.432+0000 7fd9e3a8f640 10 monclient: shutdown 2024-05-15T12:43:11.445 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.432+0000 7fd9e3a8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:11.445 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.432+0000 7fd9e3a8f640 1 -- 172.21.15.117:0/83817810 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9dc14ab70 msgr2=0x7fd9dc1514c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.445 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.432+0000 7fd9e3a8f640 1 --2- 172.21.15.117:0/83817810 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9dc14ab70 0x7fd9dc1514c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc0adb20 tx=0x7fd9d802b840 comp rx=0 tx=0).stop 2024-05-15T12:43:11.445 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.432+0000 7fd9e3a8f640 1 -- 172.21.15.117:0/83817810 shutdown_connections 2024-05-15T12:43:11.445 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.432+0000 7fd9e3a8f640 1 --2- 172.21.15.117:0/83817810 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9dc14ab70 0x7fd9dc1514c0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.445 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.432+0000 7fd9e3a8f640 1 --2- 172.21.15.117:0/83817810 >> v2:172.21.15.178:3300/0 conn(0x7fd9dc06eac0 0x7fd9dc14a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.445 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.432+0000 7fd9e3a8f640 1 -- 172.21.15.117:0/83817810 >> 172.21.15.117:0/83817810 conn(0x7fd9dc06ada0 msgr2=0x7fd9dc06b1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:11.445 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.432+0000 7fd9e3a8f640 1 -- 172.21.15.117:0/83817810 shutdown_connections 2024-05-15T12:43:11.445 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.432+0000 7fd9e3a8f640 1 -- 172.21.15.117:0/83817810 wait complete. 2024-05-15T12:43:11.446 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.432+0000 7f4e4e36d640 10 monclient(hunting): handle_auth_done global_id 4304 payload 824 2024-05-15T12:43:11.446 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.432+0000 7f4e4e36d640 1 -- 172.21.15.117:0/1274867206 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4e48144500 msgr2=0x7f4e48130e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.446 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.432+0000 7f4e4e36d640 1 --2- 172.21.15.117:0/1274867206 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4e48144500 0x7f4e48130e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.446 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.432+0000 7f4e4e36d640 1 -- 172.21.15.117:0/1274867206 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e480b3000 msgr2=0x7f4e481308e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.446 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.432+0000 7f4e4e36d640 1 --2- 172.21.15.117:0/1274867206 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e480b3000 0x7f4e481308e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.446 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.432+0000 7f4e4e36d640 10 monclient: _finish_hunting 0 2024-05-15T12:43:11.446 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.432+0000 7f4e4e36d640 1 monclient: found mon.noname-c 2024-05-15T12:43:11.446 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.432+0000 7f4e4e36d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:11.447 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.432+0000 7f4e4e36d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.117:3301/0 2024-05-15T12:43:11.447 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.432+0000 7f4e4e36d640 1 -- 172.21.15.117:0/1274867206 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e480aefc0 con 0x7f4e480b1f30 2024-05-15T12:43:11.448 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.433+0000 7fd9e3a8f640 10 monclient: build_initial_monmap 2024-05-15T12:43:11.448 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.433+0000 7fd9e3a8f640 10 monclient: monmap: 2024-05-15T12:43:11.449 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:11.449 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:43:11.449 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-05-15T12:43:11.449 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-05-15T12:43:11.449 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:11.449 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.449 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T12:43:11.449 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T12:43:11.449 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T12:43:11.449 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.449 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.433+0000 7fd9e3a8f640 1 librados: starting msgr at 2024-05-15T12:43:11.449 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.434+0000 7fd9e3a8f640 1 librados: starting objecter 2024-05-15T12:43:11.449 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.434+0000 7fd9e3a8f640 1 Processor -- start 2024-05-15T12:43:11.449 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.434+0000 7fd9e3a8f640 1 -- start start 2024-05-15T12:43:11.449 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.434+0000 7fd9e3a8f640 1 librados: setting wanted keys 2024-05-15T12:43:11.449 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.434+0000 7fd9e3a8f640 1 librados: calling monclient init 2024-05-15T12:43:11.450 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.434+0000 7fd9e3a8f640 10 monclient: init 2024-05-15T12:43:11.450 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.435+0000 7fd9e3a8f640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:11.450 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.435+0000 7fd9e3a8f640 10 monclient: _add_conns ranks=[2,1,0] 2024-05-15T12:43:11.450 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.435+0000 7fd9e3a8f640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd9dc06bd70 0x7fd9dc0ab630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.450 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.435+0000 7fd9e3a8f640 10 monclient(hunting): picked mon.noname-c con 0x7fd9dc06bd70 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:43:11.450 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.435+0000 7fd9e3a8f640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd9dc06eac0 0x7fd9dc144840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.450 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.435+0000 7fd9e3a8f640 10 monclient(hunting): picked mon.noname-b con 0x7fd9dc06eac0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:43:11.450 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.435+0000 7fd9e3a8f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9dc14ab70 0x7fd9dc144d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.450 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.435+0000 7fd9e3a8f640 10 monclient(hunting): picked mon.noname-a con 0x7fd9dc14ab70 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T12:43:11.450 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.435+0000 7fd9e3a8f640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:11.450 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.435+0000 7fd9e3a8f640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9dc06ef30 con 0x7fd9dc14ab70 2024-05-15T12:43:11.450 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.435+0000 7fd9e3a8f640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:11.450 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.435+0000 7fd9e3a8f640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9dc06e5a0 con 0x7fd9dc06eac0 2024-05-15T12:43:11.450 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.435+0000 7fd9e3a8f640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:11.450 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.435+0000 7fd9e3a8f640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9dc14afa0 con 0x7fd9dc06bd70 2024-05-15T12:43:11.451 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.435+0000 7fd9e3a8f640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:11.451 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.435+0000 7fd9e3a8f640 10 monclient(hunting): authenticate will time out at 1354.677340s 2024-05-15T12:43:11.451 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.439+0000 7f19e481c640 1 -- 172.21.15.117:0/1789255093 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3779508959 0 0) 0x7f19d802d480 con 0x7f19d8035630 2024-05-15T12:43:11.451 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.439+0000 7f19e481c640 10 monclient(hunting): _negotiate 2024-05-15T12:43:11.451 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.439+0000 7f19e481c640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:11.451 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.439+0000 7f19e481c640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:11.451 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.439+0000 7f19e481c640 10 monclient(hunting): my global_id is 4326 2024-05-15T12:43:11.451 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.439+0000 7f19e481c640 1 -- 172.21.15.117:0/1789255093 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19cc003520 con 0x7f19d8035630 2024-05-15T12:43:11.452 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.440+0000 7fd9e202c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9dc14ab70 0x7fd9dc144d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:11.452 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.441+0000 7fd9e202c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9dc14ab70 0x7fd9dc144d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:60218/0 (socket says 172.21.15.117:60218) 2024-05-15T12:43:11.452 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.441+0000 7fd9e202c640 1 -- 172.21.15.117:0/1186925958 learned_addr learned my addr 172.21.15.117:0/1186925958 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:43:11.452 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.441+0000 7fd9e202c640 10 monclient(hunting): get_auth_request con 0x7fd9dc14ab70 auth_method 0 2024-05-15T12:43:11.452 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.441+0000 7fd9e202c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:11.452 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.441+0000 7fd9e202c640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:11.452 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.441+0000 7fd9e202c640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:11.453 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.445+0000 7f19e481c640 1 -- 172.21.15.117:0/1789255093 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1716905021 0 0) 0x7f19cc003520 con 0x7f19d8035630 2024-05-15T12:43:11.453 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.445+0000 7f19e481c640 1 -- 172.21.15.117:0/1789255093 >> v2:172.21.15.178:3300/0 conn(0x7f19d80c45c0 msgr2=0x7f19d80c4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:43:11.453 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.445+0000 7f19e481c640 1 --2- 172.21.15.117:0/1789255093 >> v2:172.21.15.178:3300/0 conn(0x7f19d80c45c0 0x7f19d80c4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.454 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.445+0000 7f19e481c640 1 -- 172.21.15.117:0/1789255093 >> v1:172.21.15.178:6789/0 conn(0x7f19d8035250 legacy=0x7f19d80c1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:43:11.454 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.445+0000 7f19e481c640 10 monclient: _finish_hunting 0 2024-05-15T12:43:11.454 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.445+0000 7f19e481c640 1 monclient: found mon.noname-a-legacy 2024-05-15T12:43:11.454 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.445+0000 7f19e481c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:11.454 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.445+0000 7f19e481c640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.117:6789/0 2024-05-15T12:43:11.454 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.445+0000 7f19e481c640 1 -- 172.21.15.117:0/1789255093 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19d8032580 con 0x7f19d8035630 2024-05-15T12:43:11.454 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.445+0000 7f19e481c640 10 monclient: _finish_auth 0 2024-05-15T12:43:11.454 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.445+0000 7f19e481c640 10 monclient: _check_auth_tickets 2024-05-15T12:43:11.454 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.445+0000 7f19e481c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:11.454 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.445+0000 7f19e6820640 5 monclient: authenticate success, global_id 4326 2024-05-15T12:43:11.454 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.445+0000 7f19e6820640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:43:11.454 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.445+0000 7f19e481c640 1 -- 172.21.15.117:0/1789255093 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3166367242 0 0) 0x7f19d0001650 con 0x7f19d8035630 2024-05-15T12:43:11.454 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.445+0000 7f19e481c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:11.455 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.445+0000 7f19e481c640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-05-15T12:43:11.455 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.445+0000 7f19e481c640 10 monclient: dump: 2024-05-15T12:43:11.455 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:11.455 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:11.455 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.455 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.455 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:11.455 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.455 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:11.455 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:11.455 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:11.455 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.455 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.445+0000 7f19e481c640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] but i'm connected to v1:172.21.15.117:6789/0, reconnecting 2024-05-15T12:43:11.455 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.445+0000 7f19e481c640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:11.455 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.445+0000 7f19e481c640 1 -- 172.21.15.117:0/1789255093 >> v1:172.21.15.117:6789/0 conn(0x7f19d8035630 legacy=0x7f19d8031e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.455 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.445+0000 7f19e481c640 10 monclient: _add_conns ranks=[1,2,0] 2024-05-15T12:43:11.456 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.445+0000 7f19e481c640 1 --2- 172.21.15.117:0/1789255093 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f19cc0039d0 0x7f19cc003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.456 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.445+0000 7f19e481c640 10 monclient(hunting): picked mon.b con 0x7f19cc0039d0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:43:11.456 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.445+0000 7f19e481c640 1 --2- 172.21.15.117:0/1789255093 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f19cc004800 0x7f19cc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.456 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.445+0000 7f19e481c640 10 monclient(hunting): picked mon.c con 0x7f19cc004800 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:43:11.456 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.445+0000 7f19e481c640 1 --2- 172.21.15.117:0/1789255093 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f19cc0053c0 0x7f19cc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.456 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.445+0000 7f19e481c640 10 monclient(hunting): picked mon.a con 0x7f19cc0053c0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T12:43:11.456 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.445+0000 7f19e481c640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:11.456 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.445+0000 7f19e481c640 1 -- 172.21.15.117:0/1789255093 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19cc00c250 con 0x7f19cc0053c0 2024-05-15T12:43:11.456 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.445+0000 7f19e481c640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:11.456 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.445+0000 7f19e481c640 1 -- 172.21.15.117:0/1789255093 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19cc00c400 con 0x7f19cc0039d0 2024-05-15T12:43:11.456 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.445+0000 7f19e481c640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:11.456 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.445+0000 7f19e481c640 1 -- 172.21.15.117:0/1789255093 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19cc00c600 con 0x7f19cc004800 2024-05-15T12:43:11.456 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.445+0000 7f19e481c640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:11.456 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.445+0000 7f19e481c640 10 monclient(hunting): _finish_auth 0 2024-05-15T12:43:11.457 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.445+0000 7f19e6820640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-05-15T12:43:11.457 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.446+0000 7fd9e202c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:11.457 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.446+0000 7fd9e202c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:11.457 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.446+0000 7fd9e202c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:11.457 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.447+0000 7f19e601f640 1 --2- 172.21.15.117:0/1789255093 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f19cc0053c0 0x7f19cc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:11.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.448+0000 7f19e601f640 10 monclient(hunting): get_auth_request con 0x7f19cc0053c0 auth_method 0 2024-05-15T12:43:11.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.448+0000 7f19e601f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:11.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.448+0000 7f19e601f640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:11.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.448+0000 7f19e601f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-05-15T12:43:11.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.448+0000 7f19e601f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:11.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.448+0000 7f19e601f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:11.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.448+0000 7f19e601f640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-05-15T12:43:11.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.451+0000 7fd9e202c640 10 monclient(hunting): handle_auth_done global_id 4329 payload 824 2024-05-15T12:43:11.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.451+0000 7fd9e202c640 1 -- 172.21.15.117:0/1186925958 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd9dc06bd70 msgr2=0x7fd9dc0ab630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.451+0000 7fd9e202c640 1 --2- 172.21.15.117:0/1186925958 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd9dc06bd70 0x7fd9dc0ab630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.451+0000 7fd9e202c640 1 -- 172.21.15.117:0/1186925958 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd9dc06eac0 msgr2=0x7fd9dc144840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.451+0000 7fd9e202c640 1 --2- 172.21.15.117:0/1186925958 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd9dc06eac0 0x7fd9dc144840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.451+0000 7fd9e202c640 10 monclient: _finish_hunting 0 2024-05-15T12:43:11.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.451+0000 7fd9e202c640 1 monclient: found mon.noname-a 2024-05-15T12:43:11.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.451+0000 7fd9e202c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:11.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.451+0000 7fd9e202c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.117:3300/0 2024-05-15T12:43:11.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.451+0000 7fd9e202c640 1 -- 172.21.15.117:0/1186925958 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9dc127b10 con 0x7fd9dc14ab70 2024-05-15T12:43:11.460 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.454+0000 7f19e601f640 10 monclient(hunting): handle_auth_done global_id 4326 payload 293 2024-05-15T12:43:11.460 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.454+0000 7f19e601f640 1 -- 172.21.15.117:0/1789255093 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f19cc004800 msgr2=0x7f19cc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:43:11.460 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.454+0000 7f19e601f640 1 --2- 172.21.15.117:0/1789255093 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f19cc004800 0x7f19cc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.460 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.454+0000 7f19e601f640 1 -- 172.21.15.117:0/1789255093 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f19cc0039d0 msgr2=0x7f19cc003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:43:11.460 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.454+0000 7f19e601f640 1 --2- 172.21.15.117:0/1789255093 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f19cc0039d0 0x7f19cc003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.460 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.454+0000 7f19e601f640 10 monclient: _finish_hunting 0 2024-05-15T12:43:11.460 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.454+0000 7f19e601f640 1 monclient: found mon.a 2024-05-15T12:43:11.460 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.454+0000 7f19e601f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:11.461 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.454+0000 7f19e601f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:43:11.461 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.454+0000 7f19e601f640 1 -- 172.21.15.117:0/1789255093 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19d8032580 con 0x7f19cc0053c0 2024-05-15T12:43:11.461 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.454+0000 7f19e601f640 10 monclient: _finish_auth 0 2024-05-15T12:43:11.461 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.454+0000 7f19e601f640 10 monclient: _check_auth_tickets 2024-05-15T12:43:11.461 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.454+0000 7f19e601f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:11.461 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.454+0000 7f19e601f640 1 -- 172.21.15.117:0/1789255093 reap_dead start 2024-05-15T12:43:11.461 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.457+0000 7fd9e202c640 1 --2- 172.21.15.117:0/1186925958 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9dc14ab70 0x7fd9dc144d80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd9d8009890 tx=0x7fd9d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:11.461 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.457+0000 7fd9d27fc640 10 client.?.objecter ms_handle_connect 0x7fd9dc14ab70 2024-05-15T12:43:11.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.457+0000 7fd9d27fc640 10 client.?.objecter resend_mon_ops 2024-05-15T12:43:11.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.458+0000 7f19e601f640 1 --2- 172.21.15.117:0/1789255093 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f19cc0053c0 0x7f19cc00b9c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f19e0073c80 tx=0x7f19cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e7edbfa7b3d8950 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:11.464 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.464+0000 7f2828515640 10 monclient: get_monmap_and_config 2024-05-15T12:43:11.464 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.464+0000 7f2828515640 10 monclient: build_initial_monmap 2024-05-15T12:43:11.464 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.464+0000 7f2828515640 10 monclient: monmap: 2024-05-15T12:43:11.464 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:11.464 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:11.464 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:43:11.464485+0000 2024-05-15T12:43:11.464 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:43:11.464485+0000 2024-05-15T12:43:11.464 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:11.464 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.464 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T12:43:11.464 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T12:43:11.464 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T12:43:11.465 INFO:teuthology.orchestra.run.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T12:43:11.465 INFO:teuthology.orchestra.run.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T12:43:11.465 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.465 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.464+0000 7f2828515640 1 Processor -- start 2024-05-15T12:43:11.465 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.464+0000 7f2828515640 1 -- start start 2024-05-15T12:43:11.465 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.464+0000 7f2828515640 10 monclient: init 2024-05-15T12:43:11.465 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.464+0000 7f2828515640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:11.465 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.464+0000 7f2828515640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-05-15T12:43:11.465 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.464+0000 7f2828515640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f28200b4d00 0x7f28200b1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.465 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.464+0000 7f2828515640 10 monclient(hunting): picked mon.noname-b con 0x7f28200b4d00 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:43:11.465 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.465+0000 7f2828515640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f28200b50e0 addr v1:172.21.15.117:6789/0 2024-05-15T12:43:11.465 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.465+0000 7f2828515640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f28200a94b0 0x7f28200b0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.465 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.465+0000 7f2828515640 10 monclient(hunting): picked mon.noname-c con 0x7f28200a94b0 addr v2:172.21.15.178:3300/0 2024-05-15T12:43:11.465 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.465+0000 7f2828515640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28200ab1b0 con 0x7f28200b50e0 2024-05-15T12:43:11.465 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.465+0000 7f2828515640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:11.466 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.465+0000 7f2828515640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28200b23c0 con 0x7f28200a94b0 2024-05-15T12:43:11.466 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.465+0000 7f2828515640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:11.466 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.465+0000 7f2828515640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28200b1520 con 0x7f28200b4d00 2024-05-15T12:43:11.466 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.465+0000 7f2828515640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:11.466 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.465+0000 7f2828515640 10 monclient(hunting): authenticate will time out at 1354.707100s 2024-05-15T12:43:11.467 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.467+0000 7fd9d27fc640 1 -- 172.21.15.117:0/1186925958 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d80315a0 con 0x7fd9dc14ab70 2024-05-15T12:43:11.467 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.467+0000 7fd9d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:11.467 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.467+0000 7fd9d27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T12:43:11.467 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.467+0000 7fd9d27fc640 10 monclient: dump: 2024-05-15T12:43:11.467 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:11.467 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:11.467 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.468 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.468 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:11.468 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.468 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:11.468 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:11.468 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:11.468 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.468 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.467+0000 7fd9d27fc640 10 monclient: _finish_auth 0 2024-05-15T12:43:11.468 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.467+0000 7fd9d27fc640 10 monclient: _check_auth_tickets 2024-05-15T12:43:11.468 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.467+0000 7fd9d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:11.468 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.467+0000 7fd9d27fc640 1 -- 172.21.15.117:0/1186925958 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9d8032640 con 0x7fd9dc14ab70 2024-05-15T12:43:11.468 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.467+0000 7fd9d27fc640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:11.468 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.467+0000 7fd9d27fc640 1 -- 172.21.15.117:0/1186925958 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d8032a30 con 0x7fd9dc14ab70 2024-05-15T12:43:11.468 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.467+0000 7fd9d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:11.468 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.467+0000 7fd9d27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T12:43:11.469 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.467+0000 7fd9d27fc640 10 monclient: dump: 2024-05-15T12:43:11.469 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:11.469 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:11.469 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.469 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.469 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:11.469 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.469 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:11.469 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:11.469 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:11.469 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.469 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.467+0000 7fd9d3fff640 4 set_mon_vals no callback set 2024-05-15T12:43:11.469 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.468+0000 7fd9e3a8f640 5 monclient: authenticate success, global_id 4329 2024-05-15T12:43:11.469 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.468+0000 7fd9e3a8f640 10 monclient: _renew_subs 2024-05-15T12:43:11.469 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.468+0000 7fd9e3a8f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:43:11.469 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.468+0000 7fd9e3a8f640 1 -- 172.21.15.117:0/1186925958 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9dc125720 con 0x7fd9dc14ab70 2024-05-15T12:43:11.469 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.468+0000 7fd9e3a8f640 10 client.4329.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:43:11.470 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.468+0000 7fd9e3a8f640 10 monclient: _renew_subs 2024-05-15T12:43:11.470 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.468+0000 7fd9e3a8f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:43:11.470 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.468+0000 7fd9e3a8f640 1 -- 172.21.15.117:0/1186925958 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9dc125e90 con 0x7fd9dc14ab70 2024-05-15T12:43:11.470 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.468+0000 7fd9e3a8f640 1 librados: init done 2024-05-15T12:43:11.470 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.469+0000 7fd9e3a8f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:11.470 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.469+0000 7fd9e3a8f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:11.470 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.469+0000 7fd9e3a8f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:43:11.470 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.469+0000 7fd9e3a8f640 1 -- 172.21.15.117:0/1186925958 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9dc04f0e0 con 0x7fd9dc14ab70 2024-05-15T12:43:11.471 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.471+0000 7f19e481c640 1 -- 172.21.15.117:0/1789255093 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e00b2070 con 0x7f19cc0053c0 2024-05-15T12:43:11.471 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.471+0000 7f19e481c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:11.471 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.471+0000 7f19e481c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T12:43:11.471 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.471+0000 7f19e481c640 10 monclient: dump: 2024-05-15T12:43:11.471 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:11.471 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:11.471 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.471 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.471 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:11.471 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.471 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:11.471 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:11.471 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:11.472 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.472 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.471+0000 7f19e481c640 1 -- 172.21.15.117:0/1789255093 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19e00ad600 con 0x7f19cc0053c0 2024-05-15T12:43:11.472 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.471+0000 7f19e481c640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:11.472 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.471+0000 7f19e481c640 1 -- 172.21.15.117:0/1789255093 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e00627d0 con 0x7f19cc0053c0 2024-05-15T12:43:11.472 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.471+0000 7f19e481c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:11.472 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.471+0000 7f19e481c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T12:43:11.472 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.471+0000 7f19e481c640 10 monclient: dump: 2024-05-15T12:43:11.472 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:11.472 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:11.472 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.472 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.472 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:11.472 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.473 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:11.473 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:11.473 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:11.473 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.473 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.471+0000 7f19e6820640 10 monclient: get_monmap_and_config success 2024-05-15T12:43:11.473 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.471+0000 7f19e6820640 4 set_mon_vals no callback set 2024-05-15T12:43:11.473 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.471+0000 7f2825ab0640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f28200b50e0 0x7f2820145c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:48706/0 (socket says 172.21.15.117:48706) 2024-05-15T12:43:11.473 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.471+0000 7f2825ab0640 1 -- 172.21.15.117:0/2185971543 learned_addr learned my addr 172.21.15.117:0/2185971543 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-05-15T12:43:11.475 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.472+0000 7f19e6820640 10 monclient: shutdown 2024-05-15T12:43:11.475 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.472+0000 7f19e6820640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:11.475 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.472+0000 7f19e6820640 1 -- 172.21.15.117:0/1789255093 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f19cc0053c0 msgr2=0x7f19cc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.475 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.472+0000 7f19e6820640 1 --2- 172.21.15.117:0/1789255093 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f19cc0053c0 0x7f19cc00b9c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f19e0073c80 tx=0x7f19cc005830 comp rx=0 tx=0).stop 2024-05-15T12:43:11.475 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.472+0000 7f19e6820640 1 -- 172.21.15.117:0/1789255093 shutdown_connections 2024-05-15T12:43:11.475 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.472+0000 7f19e6820640 1 --2- 172.21.15.117:0/1789255093 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f19cc0053c0 0x7f19cc00b9c0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.475 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.472+0000 7f19e6820640 1 -- 172.21.15.117:0/1789255093 >> 172.21.15.117:0/1789255093 conn(0x7f19d8006ab0 msgr2=0x7f19d8006ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:11.475 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.472+0000 7f19e6820640 1 -- 172.21.15.117:0/1789255093 shutdown_connections 2024-05-15T12:43:11.476 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.472+0000 7f19e6820640 1 -- 172.21.15.117:0/1789255093 wait complete. 2024-05-15T12:43:11.476 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.473+0000 7f19e6820640 10 monclient: build_initial_monmap 2024-05-15T12:43:11.476 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.473+0000 7f19e6820640 10 monclient: monmap: 2024-05-15T12:43:11.476 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:11.476 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:43:11.476 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-05-15T12:43:11.476 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-05-15T12:43:11.476 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:11.476 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.476 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T12:43:11.476 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T12:43:11.476 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T12:43:11.476 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.476 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.474+0000 7f19e6820640 1 librados: starting msgr at 2024-05-15T12:43:11.476 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.474+0000 7f19e6820640 1 librados: starting objecter 2024-05-15T12:43:11.476 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.474+0000 7f19e6820640 1 Processor -- start 2024-05-15T12:43:11.477 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.474+0000 7f19e6820640 1 -- start start 2024-05-15T12:43:11.477 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.474+0000 7f19e6820640 1 librados: setting wanted keys 2024-05-15T12:43:11.477 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.474+0000 7f19e6820640 1 librados: calling monclient init 2024-05-15T12:43:11.477 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.474+0000 7f19e6820640 10 monclient: init 2024-05-15T12:43:11.477 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.475+0000 7f19e6820640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:11.477 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.475+0000 7f19e6820640 10 monclient: _add_conns ranks=[1,0,2] 2024-05-15T12:43:11.477 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.475+0000 7f19e6820640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f19cc0053c0 0x7f19d80b59e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.477 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.475+0000 7f19e6820640 10 monclient(hunting): picked mon.noname-b con 0x7f19cc0053c0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:43:11.477 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.475+0000 7f19e6820640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f19d802d8c0 0x7f19d802f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.477 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.475+0000 7f19e6820640 10 monclient(hunting): picked mon.noname-a con 0x7f19d802d8c0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T12:43:11.477 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.475+0000 7f19e6820640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f19d802fff0 0x7f19d80b2610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.477 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.475+0000 7f19e6820640 10 monclient(hunting): picked mon.noname-c con 0x7f19d802fff0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:43:11.477 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.475+0000 7f19e6820640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:11.477 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.475+0000 7f19e6820640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19d80c1a90 con 0x7f19d802d8c0 2024-05-15T12:43:11.478 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.475+0000 7f19e6820640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:11.478 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.475+0000 7f19e6820640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19d80c1910 con 0x7f19cc0053c0 2024-05-15T12:43:11.478 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.475+0000 7f19e6820640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:11.478 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.475+0000 7f19e6820640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19d80c1c10 con 0x7f19d802fff0 2024-05-15T12:43:11.478 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.475+0000 7f19e6820640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:11.478 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.475+0000 7f19e6820640 10 monclient(hunting): authenticate will time out at 1354.717243s 2024-05-15T12:43:11.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.501+0000 7f8389c6a640 10 monclient: get_monmap_and_config 2024-05-15T12:43:11.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.501+0000 7f8389c6a640 10 monclient: build_initial_monmap 2024-05-15T12:43:11.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.501+0000 7f8389c6a640 10 monclient: monmap: 2024-05-15T12:43:11.492 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:11.492 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:11.492 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:43:11.502890+0000 2024-05-15T12:43:11.492 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:43:11.502890+0000 2024-05-15T12:43:11.492 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:11.492 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.492 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T12:43:11.492 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T12:43:11.492 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T12:43:11.492 INFO:teuthology.orchestra.run.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T12:43:11.492 INFO:teuthology.orchestra.run.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T12:43:11.492 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.494 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.502+0000 7f8389c6a640 1 Processor -- start 2024-05-15T12:43:11.494 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.502+0000 7f8389c6a640 1 -- start start 2024-05-15T12:43:11.494 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.502+0000 7f8389c6a640 10 monclient: init 2024-05-15T12:43:11.495 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.503+0000 7f8389c6a640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:11.495 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.503+0000 7f8389c6a640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-05-15T12:43:11.495 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.503+0000 7f8389c6a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f83840b4d00 addr v1:172.21.15.117:6789/0 2024-05-15T12:43:11.495 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.503+0000 7f8389c6a640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f83840b50e0 0x7f83840b1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.495 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.503+0000 7f8389c6a640 10 monclient(hunting): picked mon.noname-c con 0x7f83840b50e0 addr v2:172.21.15.178:3300/0 2024-05-15T12:43:11.495 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.503+0000 7f8389c6a640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f83840a94b0 0x7f83840b0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.495 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.503+0000 7f8389c6a640 10 monclient(hunting): picked mon.noname-b con 0x7f83840a94b0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:43:11.495 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.503+0000 7f8389c6a640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83840ab1b0 con 0x7f83840b4d00 2024-05-15T12:43:11.495 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.503+0000 7f8389c6a640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:11.495 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.503+0000 7f8389c6a640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83840b23c0 con 0x7f83840b50e0 2024-05-15T12:43:11.495 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.503+0000 7f8389c6a640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:11.495 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.503+0000 7f8389c6a640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83840b1520 con 0x7f83840a94b0 2024-05-15T12:43:11.495 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.503+0000 7f8389c6a640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:11.495 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.503+0000 7f8389c6a640 10 monclient(hunting): authenticate will time out at 1354.745252s 2024-05-15T12:43:11.501 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.511+0000 7fb478ff9640 1 -- 172.21.15.117:0/1085814967 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fb46c01a070 con 0x7fb47c0a94b0 2024-05-15T12:43:11.502 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.512+0000 7fb478ff9640 1 --2- 172.21.15.117:0/1085814967 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fb46806ceb0 0x7fb46806f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.502 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.512+0000 7fb47b7fe640 1 --2- 172.21.15.117:0/1085814967 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fb46806ceb0 0x7fb46806f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:11.502 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.512+0000 7fb47b7fe640 10 monclient: get_auth_request con 0x7fb46806ceb0 auth_method 0 2024-05-15T12:43:11.503 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.513+0000 7fb478ff9640 10 client.4301.objecter ms_handle_connect 0x7fb46806ceb0 2024-05-15T12:43:11.503 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.513+0000 7fb47b7fe640 1 --2- 172.21.15.117:0/1085814967 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fb46806ceb0 0x7fb46806f370 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb47c0aed50 tx=0x7fb470008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:11.592 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.601+0000 7f9ee2ffd640 1 -- 172.21.15.117:0/723854573 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f9ee8038020 con 0x7f9eec033930 2024-05-15T12:43:11.592 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.602+0000 7f9ee2ffd640 1 --2- 172.21.15.117:0/723854573 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f9ed406ceb0 0x7f9ed406f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.592 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.602+0000 7f9ef259c640 1 --2- 172.21.15.117:0/723854573 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f9ed406ceb0 0x7f9ed406f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:11.592 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.602+0000 7f9ef259c640 10 monclient: get_auth_request con 0x7f9ed406ceb0 auth_method 0 2024-05-15T12:43:11.592 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.602+0000 7f9ef259c640 1 --2- 172.21.15.117:0/723854573 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f9ed406ceb0 0x7f9ed406f370 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9ee40010e0 tx=0x7f9ee4007be0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:11.592 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.602+0000 7f9ee2ffd640 10 client.4348.objecter ms_handle_connect 0x7f9ed406ceb0 2024-05-15T12:43:11.595 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.604+0000 7f9ee2ffd640 1 -- 172.21.15.117:0/723854573 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f9ee80792b0 con 0x7f9eec033930 2024-05-15T12:43:11.595 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.604+0000 7f9ee2ffd640 10 client.4348.objecter ms_dispatch 0x7f9ef4000b90 osd_map(26..26 src has 1..26) v4 2024-05-15T12:43:11.595 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.604+0000 7f9ee2ffd640 3 client.4348.objecter handle_osd_map got epochs [26,26] > 0 2024-05-15T12:43:11.595 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.604+0000 7f9ee2ffd640 3 client.4348.objecter handle_osd_map decoding full epoch 26 2024-05-15T12:43:11.595 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.604+0000 7f9ee2ffd640 20 client.4348.objecter dump_active .. 0 homeless 2024-05-15T12:43:11.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.615+0000 7f9ee2ffd640 1 -- 172.21.15.117:0/723854573 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f9eec0c2280 con 0x7f9eec033930 2024-05-15T12:43:11.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.615+0000 7f9ee2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:11.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.615+0000 7f9ee2ffd640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:43:11.635 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.645+0000 7f19e581e640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f19cc0053c0 0x7f19d80b59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:11.636 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.646+0000 7f2826ab2640 1 --2- 172.21.15.117:0/2185971543 >> v2:172.21.15.178:3300/0 conn(0x7f28200a94b0 0x7f28200b0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:11.636 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.646+0000 7f19e581e640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f19cc0053c0 0x7f19d80b59e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:42554/0 (socket says 172.21.15.117:42554) 2024-05-15T12:43:11.636 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.646+0000 7f19e581e640 1 -- 172.21.15.117:0/3224956099 learned_addr learned my addr 172.21.15.117:0/3224956099 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:43:11.636 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.646+0000 7f19e581e640 10 monclient(hunting): get_auth_request con 0x7f19cc0053c0 auth_method 0 2024-05-15T12:43:11.637 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.646+0000 7f19e581e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:11.637 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.646+0000 7f19e581e640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:11.637 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.646+0000 7f19e581e640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:11.637 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.647+0000 7f2826ab2640 10 monclient(hunting): get_auth_request con 0x7f28200a94b0 auth_method 0 2024-05-15T12:43:11.637 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.647+0000 7f2826ab2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:11.637 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.647+0000 7f2826ab2640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:11.637 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.647+0000 7f2826ab2640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:11.638 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.647+0000 7f19e581e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:11.638 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.647+0000 7f19e581e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:11.638 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.647+0000 7f19e581e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:11.640 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.650+0000 7f19e581e640 10 monclient(hunting): handle_auth_done global_id 4351 payload 824 2024-05-15T12:43:11.641 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.650+0000 7f19e581e640 1 -- 172.21.15.117:0/3224956099 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f19d802fff0 msgr2=0x7f19d80b2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.641 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.650+0000 7f19e581e640 1 --2- 172.21.15.117:0/3224956099 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f19d802fff0 0x7f19d80b2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.641 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.650+0000 7f19e581e640 1 -- 172.21.15.117:0/3224956099 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f19d802d8c0 msgr2=0x7f19d802f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.641 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.650+0000 7f19e581e640 1 --2- 172.21.15.117:0/3224956099 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f19d802d8c0 0x7f19d802f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.641 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.650+0000 7f19e581e640 10 monclient: _finish_hunting 0 2024-05-15T12:43:11.641 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.650+0000 7f19e581e640 1 monclient: found mon.noname-b 2024-05-15T12:43:11.641 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.650+0000 7f19e581e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:11.641 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.650+0000 7f19e581e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2024-05-15T12:43:11.641 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.650+0000 7f19e581e640 1 -- 172.21.15.117:0/3224956099 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19d802a7d0 con 0x7f19cc0053c0 2024-05-15T12:43:11.642 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.652+0000 7f2826ab2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:11.642 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.652+0000 7f2826ab2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:11.642 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.652+0000 7f2826ab2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:11.642 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.652+0000 7f8382ffd640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f83840b50e0 0x7f83840b1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:11.647 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.657+0000 7f8382ffd640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f83840b50e0 0x7f83840b1e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:42566/0 (socket says 172.21.15.117:42566) 2024-05-15T12:43:11.647 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.657+0000 7f8382ffd640 1 -- 172.21.15.117:0/3884409320 learned_addr learned my addr 172.21.15.117:0/3884409320 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:43:11.647 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.657+0000 7f8382ffd640 10 monclient(hunting): get_auth_request con 0x7f83840b50e0 auth_method 0 2024-05-15T12:43:11.647 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.657+0000 7f8382ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:11.647 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.657+0000 7f8382ffd640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:11.647 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.657+0000 7f8382ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:11.648 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.658+0000 7f8382ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:11.648 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.658+0000 7f8382ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:11.648 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.658+0000 7f8382ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:11.650 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.659+0000 7f8382ffd640 10 monclient(hunting): handle_auth_done global_id 4357 payload 274 2024-05-15T12:43:11.650 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.659+0000 7f8382ffd640 1 -- 172.21.15.117:0/3884409320 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f83840a94b0 msgr2=0x7f83840b0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:43:11.650 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.659+0000 7f8382ffd640 1 --2- 172.21.15.117:0/3884409320 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f83840a94b0 0x7f83840b0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.650 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.659+0000 7f8382ffd640 1 -- 172.21.15.117:0/3884409320 >> v1:172.21.15.117:6789/0 conn(0x7f83840b4d00 legacy=0x7f8384145c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:43:11.650 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.659+0000 7f8382ffd640 10 monclient: _finish_hunting 0 2024-05-15T12:43:11.650 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.659+0000 7f8382ffd640 1 monclient: found mon.noname-c 2024-05-15T12:43:11.650 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.659+0000 7f8382ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:11.650 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.659+0000 7f8382ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.178:3300/0 2024-05-15T12:43:11.650 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.659+0000 7f8382ffd640 1 -- 172.21.15.117:0/3884409320 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83840b16d0 con 0x7f83840b50e0 2024-05-15T12:43:11.651 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.660+0000 7f19e581e640 1 --2- 172.21.15.117:0/3224956099 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f19cc0053c0 0x7f19d80b59e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f19dc00c440 tx=0x7f19dc00c900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:11.651 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.660+0000 7f19d67fc640 10 client.?.objecter ms_handle_connect 0x7f19cc0053c0 2024-05-15T12:43:11.651 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.660+0000 7f19d67fc640 10 client.?.objecter resend_mon_ops 2024-05-15T12:43:11.653 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.663+0000 7f2826ab2640 10 monclient(hunting): handle_auth_done global_id 4354 payload 274 2024-05-15T12:43:11.653 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.663+0000 7f2826ab2640 1 -- 172.21.15.117:0/2185971543 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f28200b4d00 msgr2=0x7f28200b1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:43:11.653 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.663+0000 7f2826ab2640 1 --2- 172.21.15.117:0/2185971543 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f28200b4d00 0x7f28200b1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.653 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.663+0000 7f2826ab2640 1 -- 172.21.15.117:0/2185971543 >> v1:172.21.15.117:6789/0 conn(0x7f28200b50e0 legacy=0x7f2820145c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.653 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.663+0000 7f2826ab2640 10 monclient: _finish_hunting 0 2024-05-15T12:43:11.653 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.663+0000 7f2826ab2640 1 monclient: found mon.noname-c 2024-05-15T12:43:11.653 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.663+0000 7f2826ab2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:11.654 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.663+0000 7f2826ab2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.178:3300/0 2024-05-15T12:43:11.654 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.663+0000 7f2826ab2640 1 -- 172.21.15.117:0/2185971543 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28200b16d0 con 0x7f28200a94b0 2024-05-15T12:43:11.657 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.667+0000 7f2826ab2640 1 --2- 172.21.15.117:0/2185971543 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f28200a94b0 0x7f28200b0fe0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2818009d90 tx=0x7f281802be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=765a1044cc5dc497 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:11.663 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.672+0000 7f28252af640 1 -- 172.21.15.117:0/2185971543 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2818039070 con 0x7f28200a94b0 2024-05-15T12:43:11.663 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.672+0000 7f28252af640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:11.663 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.672+0000 7f28252af640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:43:11.663 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.672+0000 7f28252af640 10 monclient: dump: 2024-05-15T12:43:11.663 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:11.663 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:11.663 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.663 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.663 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:11.663 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.663 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:11.664 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:11.664 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:11.664 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.672+0000 7f28252af640 10 monclient: _finish_auth 0 2024-05-15T12:43:11.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.672+0000 7f28252af640 10 monclient: _check_auth_tickets 2024-05-15T12:43:11.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.672+0000 7f28252af640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:11.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.672+0000 7f28252af640 1 -- 172.21.15.117:0/2185971543 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f281802cbc0 con 0x7f28200a94b0 2024-05-15T12:43:11.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.672+0000 7f2828515640 5 monclient: authenticate success, global_id 4354 2024-05-15T12:43:11.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.672+0000 7f28252af640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:11.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.672+0000 7f28252af640 1 -- 172.21.15.117:0/2185971543 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2818034440 con 0x7f28200a94b0 2024-05-15T12:43:11.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.672+0000 7f28252af640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:11.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.672+0000 7f28252af640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:43:11.665 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.672+0000 7f28252af640 10 monclient: dump: 2024-05-15T12:43:11.665 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:11.665 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:11.665 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.665 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.665 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:11.665 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.665 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:11.665 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:11.665 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:11.665 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.665 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.672+0000 7f2828515640 10 monclient: get_monmap_and_config success 2024-05-15T12:43:11.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.672+0000 7f2828515640 4 set_mon_vals no callback set 2024-05-15T12:43:11.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.673+0000 7f2828515640 10 monclient: shutdown 2024-05-15T12:43:11.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.673+0000 7f2828515640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:11.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.673+0000 7f2828515640 1 -- 172.21.15.117:0/2185971543 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f28200a94b0 msgr2=0x7f28200b0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.673+0000 7f2828515640 1 --2- 172.21.15.117:0/2185971543 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f28200a94b0 0x7f28200b0fe0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2818009d90 tx=0x7f281802be10 comp rx=0 tx=0).stop 2024-05-15T12:43:11.667 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.673+0000 7f8382ffd640 1 --2- 172.21.15.117:0/3884409320 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f83840b50e0 0x7f83840b1e80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8378009870 tx=0x7f837802b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69393832324d88a1 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:11.667 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.673+0000 7f2828515640 1 -- 172.21.15.117:0/2185971543 shutdown_connections 2024-05-15T12:43:11.667 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.673+0000 7f2828515640 1 --2- 172.21.15.117:0/2185971543 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f28200a94b0 0x7f28200b0fe0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.668 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.673+0000 7f2828515640 1 --2- 172.21.15.117:0/2185971543 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f28200b4d00 0x7f28200b1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.668 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.673+0000 7f2828515640 1 -- 172.21.15.117:0/2185971543 >> 172.21.15.117:0/2185971543 conn(0x7f2820066cc0 msgr2=0x7f28200670d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:11.668 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.673+0000 7f2828515640 1 -- 172.21.15.117:0/2185971543 shutdown_connections 2024-05-15T12:43:11.668 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.673+0000 7f2828515640 1 -- 172.21.15.117:0/2185971543 wait complete. 2024-05-15T12:43:11.668 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.674+0000 7f2828515640 10 monclient: build_initial_monmap 2024-05-15T12:43:11.668 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.674+0000 7f2828515640 10 monclient: monmap: 2024-05-15T12:43:11.668 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:11.668 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:43:11.668 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-05-15T12:43:11.668 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-05-15T12:43:11.668 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:11.668 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.668 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T12:43:11.668 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T12:43:11.668 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T12:43:11.669 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.669 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.674+0000 7f19d67fc640 1 -- 172.21.15.117:0/3224956099 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19dc017070 con 0x7f19cc0053c0 2024-05-15T12:43:11.669 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.674+0000 7f19d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:11.670 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.674+0000 7f19d67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:43:11.670 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.674+0000 7f19d67fc640 10 monclient: dump: 2024-05-15T12:43:11.670 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:11.670 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:11.670 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.670 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.670 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:11.670 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.670 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:11.670 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:11.670 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:11.670 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.670 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.674+0000 7f19d67fc640 10 monclient: _finish_auth 0 2024-05-15T12:43:11.670 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.674+0000 7f19d67fc640 10 monclient: _check_auth_tickets 2024-05-15T12:43:11.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.674+0000 7f19d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:11.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.674+0000 7f19d67fc640 1 -- 172.21.15.117:0/3224956099 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19dc0035f0 con 0x7f19cc0053c0 2024-05-15T12:43:11.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.674+0000 7f19d67fc640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:11.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.674+0000 7f19d67fc640 1 -- 172.21.15.117:0/3224956099 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19dc0039e0 con 0x7f19cc0053c0 2024-05-15T12:43:11.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.674+0000 7f19d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:11.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.674+0000 7f19d67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:43:11.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.674+0000 7f19d67fc640 10 monclient: dump: 2024-05-15T12:43:11.671 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:11.671 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:11.671 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.671 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.671 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:11.671 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.671 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:11.672 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:11.672 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:11.672 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.672 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.674+0000 7f19d7fff640 4 set_mon_vals no callback set 2024-05-15T12:43:11.672 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.674+0000 7f19e6820640 5 monclient: authenticate success, global_id 4351 2024-05-15T12:43:11.672 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.674+0000 7f19e6820640 10 monclient: _renew_subs 2024-05-15T12:43:11.672 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.674+0000 7f19e6820640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:43:11.672 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.674+0000 7f19e6820640 1 -- 172.21.15.117:0/3224956099 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19d802aab0 con 0x7f19cc0053c0 2024-05-15T12:43:11.672 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.674+0000 7f19e6820640 10 client.4351.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:43:11.672 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.674+0000 7f19e6820640 10 monclient: _renew_subs 2024-05-15T12:43:11.672 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.674+0000 7f19e6820640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:43:11.672 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.674+0000 7f19e6820640 1 -- 172.21.15.117:0/3224956099 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19d8030650 con 0x7f19cc0053c0 2024-05-15T12:43:11.672 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.674+0000 7f19e6820640 1 librados: init done 2024-05-15T12:43:11.674 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.674+0000 7f2828515640 1 librados: starting msgr at 2024-05-15T12:43:11.674 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.674+0000 7f2828515640 1 librados: starting objecter 2024-05-15T12:43:11.674 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.674+0000 7f2828515640 1 Processor -- start 2024-05-15T12:43:11.674 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.675+0000 7f2828515640 1 -- start start 2024-05-15T12:43:11.674 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.675+0000 7f2828515640 1 librados: setting wanted keys 2024-05-15T12:43:11.674 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.675+0000 7f2828515640 1 librados: calling monclient init 2024-05-15T12:43:11.674 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.675+0000 7f2828515640 10 monclient: init 2024-05-15T12:43:11.674 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.675+0000 7f2828515640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:11.674 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.675+0000 7f2828515640 10 monclient: _add_conns ranks=[0,2,1] 2024-05-15T12:43:11.674 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.675+0000 7f2828515640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f28200a94b0 0x7f282013e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.674 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.675+0000 7f2828515640 10 monclient(hunting): picked mon.noname-a con 0x7f28200a94b0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T12:43:11.675 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.675+0000 7f2828515640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f28200b4d00 0x7f2820142580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.675 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.675+0000 7f2828515640 10 monclient(hunting): picked mon.noname-c con 0x7f28200b4d00 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:43:11.675 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.675+0000 7f2828515640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f28201d8380 0x7f2820142c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.675 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.675+0000 7f2828515640 10 monclient(hunting): picked mon.noname-b con 0x7f28201d8380 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:43:11.675 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.675+0000 7f2828515640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:11.675 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.675+0000 7f2828515640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f282014cfb0 con 0x7f28200a94b0 2024-05-15T12:43:11.675 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.675+0000 7f2828515640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:11.675 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.675+0000 7f2828515640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f282014d130 con 0x7f28201d8380 2024-05-15T12:43:11.675 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.675+0000 7f2828515640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:11.675 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.675+0000 7f2828515640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f282014d2b0 con 0x7f28200b4d00 2024-05-15T12:43:11.675 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.675+0000 7f2828515640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:11.675 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.675+0000 7f2828515640 10 monclient(hunting): authenticate will time out at 1354.917567s 2024-05-15T12:43:11.676 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.675+0000 7f19e6820640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:11.676 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.675+0000 7f19e6820640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:11.676 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.675+0000 7f19e6820640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:43:11.676 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.675+0000 7f19e6820640 1 -- 172.21.15.117:0/3224956099 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19d802fb20 con 0x7f19cc0053c0 2024-05-15T12:43:11.685 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.695+0000 7f19d67fc640 1 -- 172.21.15.117:0/3224956099 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f19dc003b80 con 0x7f19cc0053c0 2024-05-15T12:43:11.686 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.696+0000 7f19d67fc640 1 --2- 172.21.15.117:0/3224956099 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f19c406ce60 0x7f19c406f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.696+0000 7f19e501d640 1 --2- 172.21.15.117:0/3224956099 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f19c406ce60 0x7f19c406f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:11.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.696+0000 7f19e501d640 10 monclient: get_auth_request con 0x7f19c406ce60 auth_method 0 2024-05-15T12:43:11.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.697+0000 7f19d67fc640 10 client.4351.objecter ms_handle_connect 0x7f19c406ce60 2024-05-15T12:43:11.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.697+0000 7f19e501d640 1 --2- 172.21.15.117:0/3224956099 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f19c406ce60 0x7f19c406f320 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f19d0000f60 tx=0x7f19d00014f0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:11.690 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.700+0000 7f19d67fc640 1 -- 172.21.15.117:0/3224956099 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f19dc055ea0 con 0x7f19cc0053c0 2024-05-15T12:43:11.690 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.700+0000 7f19d67fc640 10 client.4351.objecter ms_dispatch 0x7f19e0000b90 osd_map(26..26 src has 1..26) v4 2024-05-15T12:43:11.690 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.700+0000 7f19d67fc640 3 client.4351.objecter handle_osd_map got epochs [26,26] > 0 2024-05-15T12:43:11.690 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.700+0000 7f19d67fc640 3 client.4351.objecter handle_osd_map decoding full epoch 26 2024-05-15T12:43:11.690 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.700+0000 7f19d67fc640 20 client.4351.objecter dump_active .. 0 homeless 2024-05-15T12:43:11.692 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.702+0000 7f83827fc640 1 -- 172.21.15.117:0/3884409320 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83780317c0 con 0x7f83840b50e0 2024-05-15T12:43:11.692 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.702+0000 7f83827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:11.692 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.702+0000 7f83827fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:43:11.692 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.702+0000 7f83827fc640 10 monclient: dump: 2024-05-15T12:43:11.693 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:11.693 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:11.693 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.693 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.693 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:11.693 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.693 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:11.693 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:11.693 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:11.693 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.693 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.702+0000 7f83827fc640 10 monclient: _finish_auth 0 2024-05-15T12:43:11.693 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.702+0000 7f83827fc640 10 monclient: _check_auth_tickets 2024-05-15T12:43:11.693 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.702+0000 7f83827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:11.693 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.702+0000 7f83827fc640 1 -- 172.21.15.117:0/3884409320 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8378037070 con 0x7f83840b50e0 2024-05-15T12:43:11.694 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.702+0000 7f83827fc640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:11.694 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.702+0000 7f8389c6a640 5 monclient: authenticate success, global_id 4357 2024-05-15T12:43:11.694 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.702+0000 7f83827fc640 1 -- 172.21.15.117:0/3884409320 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8378031b80 con 0x7f83840b50e0 2024-05-15T12:43:11.694 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.702+0000 7f83827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:11.694 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.702+0000 7f83827fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:43:11.694 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.702+0000 7f83827fc640 10 monclient: dump: 2024-05-15T12:43:11.694 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:11.694 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:11.694 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.694 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.694 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:11.694 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.694 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:11.695 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:11.695 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:11.695 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.695 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.702+0000 7f8389c6a640 10 monclient: get_monmap_and_config success 2024-05-15T12:43:11.695 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.702+0000 7f8389c6a640 4 set_mon_vals no callback set 2024-05-15T12:43:11.695 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.702+0000 7f2826ab2640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f28201d8380 0x7f2820142c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:11.697 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.703+0000 7f8389c6a640 10 monclient: shutdown 2024-05-15T12:43:11.697 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.703+0000 7f8389c6a640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:11.697 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.703+0000 7f8389c6a640 1 -- 172.21.15.117:0/3884409320 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f83840b50e0 msgr2=0x7f83840b1e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.698 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.703+0000 7f8389c6a640 1 --2- 172.21.15.117:0/3884409320 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f83840b50e0 0x7f83840b1e80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8378009870 tx=0x7f837802b860 comp rx=0 tx=0).stop 2024-05-15T12:43:11.698 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.703+0000 7f8389c6a640 1 -- 172.21.15.117:0/3884409320 shutdown_connections 2024-05-15T12:43:11.698 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.703+0000 7f8389c6a640 1 --2- 172.21.15.117:0/3884409320 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f83840a94b0 0x7f83840b0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.698 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.703+0000 7f8389c6a640 1 --2- 172.21.15.117:0/3884409320 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f83840b50e0 0x7f83840b1e80 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.698 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.703+0000 7f8389c6a640 1 -- 172.21.15.117:0/3884409320 >> 172.21.15.117:0/3884409320 conn(0x7f8384066cc0 msgr2=0x7f83840670d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:11.698 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.703+0000 7f8389c6a640 1 -- 172.21.15.117:0/3884409320 shutdown_connections 2024-05-15T12:43:11.698 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.703+0000 7f8389c6a640 1 -- 172.21.15.117:0/3884409320 wait complete. 2024-05-15T12:43:11.698 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.704+0000 7f8389c6a640 10 monclient: build_initial_monmap 2024-05-15T12:43:11.698 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.704+0000 7f8389c6a640 10 monclient: monmap: 2024-05-15T12:43:11.698 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:11.698 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:43:11.698 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-05-15T12:43:11.698 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-05-15T12:43:11.699 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:11.699 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.699 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T12:43:11.699 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T12:43:11.699 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T12:43:11.699 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.699 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.704+0000 7f8389c6a640 1 librados: starting msgr at 2024-05-15T12:43:11.699 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.704+0000 7f8389c6a640 1 librados: starting objecter 2024-05-15T12:43:11.699 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.705+0000 7f8389c6a640 1 Processor -- start 2024-05-15T12:43:11.699 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.705+0000 7f8389c6a640 1 -- start start 2024-05-15T12:43:11.699 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.705+0000 7f8389c6a640 1 librados: setting wanted keys 2024-05-15T12:43:11.699 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.705+0000 7f8389c6a640 1 librados: calling monclient init 2024-05-15T12:43:11.699 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.705+0000 7f8389c6a640 10 monclient: init 2024-05-15T12:43:11.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.705+0000 7f8389c6a640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:11.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.705+0000 7f8389c6a640 10 monclient: _add_conns ranks=[1,0,2] 2024-05-15T12:43:11.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.705+0000 7f8389c6a640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f83840a94b0 0x7f8384142f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.705+0000 7f8389c6a640 10 monclient(hunting): picked mon.noname-b con 0x7f83840a94b0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:43:11.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.705+0000 7f8389c6a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f83840b4d00 0x7f838413e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.705+0000 7f8389c6a640 10 monclient(hunting): picked mon.noname-a con 0x7f83840b4d00 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T12:43:11.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.705+0000 7f8389c6a640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f83841db630 0x7f838413eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.705+0000 7f8389c6a640 10 monclient(hunting): picked mon.noname-c con 0x7f83841db630 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:43:11.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.705+0000 7f8389c6a640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:11.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.705+0000 7f8389c6a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f838414d130 con 0x7f83840b4d00 2024-05-15T12:43:11.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.705+0000 7f8389c6a640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:11.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.705+0000 7f8389c6a640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f838414cfb0 con 0x7f83840a94b0 2024-05-15T12:43:11.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.705+0000 7f8389c6a640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:11.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.705+0000 7f8389c6a640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f838414d2b0 con 0x7f83841db630 2024-05-15T12:43:11.701 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.705+0000 7f8389c6a640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:11.701 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.705+0000 7f8389c6a640 10 monclient(hunting): authenticate will time out at 1354.947633s 2024-05-15T12:43:11.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.716+0000 7fb478ff9640 1 -- 172.21.15.117:0/1085814967 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fb46c056310 con 0x7fb47c0a94b0 2024-05-15T12:43:11.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.716+0000 7fb478ff9640 10 client.4301.objecter ms_dispatch 0x7fb47c000b90 osd_map(26..26 src has 1..26) v4 2024-05-15T12:43:11.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.716+0000 7fb478ff9640 3 client.4301.objecter handle_osd_map got epochs [26,26] > 0 2024-05-15T12:43:11.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.716+0000 7fb478ff9640 3 client.4301.objecter handle_osd_map decoding full epoch 26 2024-05-15T12:43:11.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.716+0000 7fb478ff9640 20 client.4301.objecter dump_active .. 0 homeless 2024-05-15T12:43:11.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.716+0000 7fb478ff9640 1 -- 172.21.15.117:0/1085814967 <== mon.2 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fb46c021020 con 0x7fb47c0a94b0 2024-05-15T12:43:11.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.716+0000 7fb478ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:11.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.716+0000 7fb478ff9640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:43:11.720 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.730+0000 7fede3fff640 1 -- 172.21.15.117:0/1283422087 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fedf809c3f0 con 0x7fedf00c64a0 2024-05-15T12:43:11.720 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.730+0000 7fede3fff640 1 --2- 172.21.15.117:0/1283422087 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7feddc06ceb0 0x7feddc06f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.731+0000 7fedf77fe640 1 --2- 172.21.15.117:0/1283422087 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7feddc06ceb0 0x7feddc06f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:11.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.731+0000 7fedf77fe640 10 monclient: get_auth_request con 0x7feddc06ceb0 auth_method 0 2024-05-15T12:43:11.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.731+0000 7fedf77fe640 1 --2- 172.21.15.117:0/1283422087 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7feddc06ceb0 0x7feddc06f370 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fedec034000 tx=0x7fedec008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:11.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.731+0000 7fede3fff640 10 client.4298.objecter ms_handle_connect 0x7feddc06ceb0 2024-05-15T12:43:11.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.733+0000 7fd9d27fc640 1 -- 172.21.15.117:0/1186925958 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fd9d8032bd0 con 0x7fd9dc14ab70 2024-05-15T12:43:11.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.733+0000 7fd9d27fc640 1 --2- 172.21.15.117:0/1186925958 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fd9c006ce60 0x7fd9c006f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:11.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.733+0000 7fd9e182b640 1 --2- 172.21.15.117:0/1186925958 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fd9c006ce60 0x7fd9c006f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:11.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.733+0000 7fd9e182b640 10 monclient: get_auth_request con 0x7fd9c006ce60 auth_method 0 2024-05-15T12:43:11.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.734+0000 7fd9d27fc640 10 client.4329.objecter ms_handle_connect 0x7fd9c006ce60 2024-05-15T12:43:11.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.734+0000 7fd9e182b640 1 --2- 172.21.15.117:0/1186925958 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fd9c006ce60 0x7fd9c006f320 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc0011e0 tx=0x7fd9cc007890 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:11.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.740+0000 7f4e4e36d640 1 --2- 172.21.15.117:0/1274867206 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4e480b1f30 0x7f4e480afa30 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4e3800c410 tx=0x7f4e3800c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:11.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.740+0000 7f4e3effd640 10 client.?.objecter ms_handle_connect 0x7f4e480b1f30 2024-05-15T12:43:11.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.740+0000 7f4e3effd640 10 client.?.objecter resend_mon_ops 2024-05-15T12:43:11.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.750+0000 7fd9d27fc640 1 -- 172.21.15.117:0/1186925958 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fd9d8078b20 con 0x7fd9dc14ab70 2024-05-15T12:43:11.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.750+0000 7fd9d27fc640 10 client.4329.objecter ms_dispatch 0x7fd9dc000b90 osd_map(26..26 src has 1..26) v4 2024-05-15T12:43:11.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.750+0000 7fd9d27fc640 3 client.4329.objecter handle_osd_map got epochs [26,26] > 0 2024-05-15T12:43:11.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.750+0000 7fd9d27fc640 3 client.4329.objecter handle_osd_map decoding full epoch 26 2024-05-15T12:43:11.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.750+0000 7fd9d27fc640 20 client.4329.objecter dump_active .. 0 homeless 2024-05-15T12:43:11.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.750+0000 7fd9d27fc640 1 -- 172.21.15.117:0/1186925958 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fd9d8053080 con 0x7fd9dc14ab70 2024-05-15T12:43:11.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.750+0000 7fd9d27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:11.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.750+0000 7fd9d27fc640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:43:11.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.793+0000 7f28262b1640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f28200a94b0 0x7f282013e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:11.784 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.794+0000 7f28262b1640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f28200a94b0 0x7f282013e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:60240/0 (socket says 172.21.15.117:60240) 2024-05-15T12:43:11.784 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.794+0000 7f28262b1640 1 -- 172.21.15.117:0/1602088070 learned_addr learned my addr 172.21.15.117:0/1602088070 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:43:11.784 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.794+0000 7f28262b1640 10 monclient(hunting): get_auth_request con 0x7f28200a94b0 auth_method 0 2024-05-15T12:43:11.784 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.794+0000 7f28262b1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:11.784 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.794+0000 7f28262b1640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:11.784 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.794+0000 7f28262b1640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:11.787 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.797+0000 7f28262b1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:11.788 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.797+0000 7f28262b1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:11.788 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.797+0000 7f28262b1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:11.793 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.803+0000 7f28262b1640 10 monclient(hunting): handle_auth_done global_id 4335 payload 824 2024-05-15T12:43:11.794 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.803+0000 7f28262b1640 1 -- 172.21.15.117:0/1602088070 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f28200b4d00 msgr2=0x7f2820142580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.794 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.803+0000 7f28262b1640 1 --2- 172.21.15.117:0/1602088070 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f28200b4d00 0x7f2820142580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.794 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.803+0000 7f28262b1640 1 -- 172.21.15.117:0/1602088070 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f28201d8380 msgr2=0x7f2820142c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.794 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.803+0000 7f28262b1640 1 --2- 172.21.15.117:0/1602088070 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f28201d8380 0x7f2820142c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.794 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.803+0000 7f28262b1640 10 monclient: _finish_hunting 0 2024-05-15T12:43:11.794 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.803+0000 7f28262b1640 1 monclient: found mon.noname-a 2024-05-15T12:43:11.794 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.803+0000 7f28262b1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:11.794 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.803+0000 7f28262b1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.117:3300/0 2024-05-15T12:43:11.794 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.803+0000 7f28262b1640 1 -- 172.21.15.117:0/1602088070 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28200b0880 con 0x7f28200a94b0 2024-05-15T12:43:11.795 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.803+0000 7f8382ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f83840b4d00 0x7f838413e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:11.795 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.804+0000 7f8382ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f83840b4d00 0x7f838413e0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:60252/0 (socket says 172.21.15.117:60252) 2024-05-15T12:43:11.795 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.804+0000 7f8382ffd640 1 -- 172.21.15.117:0/568609185 learned_addr learned my addr 172.21.15.117:0/568609185 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:43:11.795 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.804+0000 7f8382ffd640 10 monclient(hunting): get_auth_request con 0x7f83840b4d00 auth_method 0 2024-05-15T12:43:11.795 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.804+0000 7f8382ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:11.795 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.804+0000 7f8382ffd640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:11.795 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.804+0000 7f8382ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:11.800 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.810+0000 7f8382ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:11.800 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.810+0000 7f8382ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:11.800 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.810+0000 7f8382ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:11.806 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.816+0000 7f8382ffd640 10 monclient(hunting): handle_auth_done global_id 4338 payload 824 2024-05-15T12:43:11.807 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.816+0000 7f8382ffd640 1 -- 172.21.15.117:0/568609185 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f83841db630 msgr2=0x7f838413eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.807 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.816+0000 7f8382ffd640 1 --2- 172.21.15.117:0/568609185 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f83841db630 0x7f838413eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.807 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.816+0000 7f8382ffd640 1 -- 172.21.15.117:0/568609185 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f83840a94b0 msgr2=0x7f8384142f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.807 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.816+0000 7f8382ffd640 1 --2- 172.21.15.117:0/568609185 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f83840a94b0 0x7f8384142f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.807 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.816+0000 7f8382ffd640 10 monclient: _finish_hunting 0 2024-05-15T12:43:11.807 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.816+0000 7f8382ffd640 1 monclient: found mon.noname-a 2024-05-15T12:43:11.807 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.816+0000 7f8382ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:11.807 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.816+0000 7f8382ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.117:3300/0 2024-05-15T12:43:11.807 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.816+0000 7f8382ffd640 1 -- 172.21.15.117:0/568609185 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83840ac230 con 0x7f83840b4d00 2024-05-15T12:43:11.807 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.816+0000 7f28262b1640 1 --2- 172.21.15.117:0/1602088070 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f28200a94b0 0x7f282013e810 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f281c00b4a0 tx=0x7f281c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:11.808 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.817+0000 7f2816ffd640 10 client.?.objecter ms_handle_connect 0x7f28200a94b0 2024-05-15T12:43:11.808 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.817+0000 7f2816ffd640 10 client.?.objecter resend_mon_ops 2024-05-15T12:43:11.809 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.819+0000 7f9ef359e640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-05-15T12:43:11.809 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.819+0000 7f9ef359e640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-05-15T12:43:11.809 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.819+0000 7f9ef359e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:43:11.809 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.819+0000 7f9ef359e640 1 -- 172.21.15.117:0/723854573 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f9eec0c8820 con 0x7f9eec033930 2024-05-15T12:43:11.816 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.825+0000 7f2816ffd640 1 -- 172.21.15.117:0/1602088070 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f281c014070 con 0x7f28200a94b0 2024-05-15T12:43:11.816 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.825+0000 7f2816ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:11.816 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.825+0000 7f2816ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T12:43:11.816 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.825+0000 7f2816ffd640 10 monclient: dump: 2024-05-15T12:43:11.816 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:11.816 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:11.816 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.816 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.816 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:11.816 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.816 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:11.817 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:11.817 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:11.817 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.817 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.825+0000 7f2816ffd640 10 monclient: _finish_auth 0 2024-05-15T12:43:11.817 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.825+0000 7f2816ffd640 10 monclient: _check_auth_tickets 2024-05-15T12:43:11.817 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.825+0000 7f2816ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:11.817 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.825+0000 7f2816ffd640 1 -- 172.21.15.117:0/1602088070 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f281c010610 con 0x7f28200a94b0 2024-05-15T12:43:11.817 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.825+0000 7f2828515640 5 monclient: authenticate success, global_id 4335 2024-05-15T12:43:11.817 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.825+0000 7f2828515640 10 monclient: _renew_subs 2024-05-15T12:43:11.817 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.825+0000 7f2828515640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:43:11.817 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.825+0000 7f2828515640 1 -- 172.21.15.117:0/1602088070 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2820143160 con 0x7f28200a94b0 2024-05-15T12:43:11.817 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.825+0000 7f2816ffd640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:11.817 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.825+0000 7f2816ffd640 1 -- 172.21.15.117:0/1602088070 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f281c010a00 con 0x7f28200a94b0 2024-05-15T12:43:11.817 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.825+0000 7f2816ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:11.817 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.825+0000 7f2828515640 10 client.4335.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:43:11.818 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.825+0000 7f2816ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T12:43:11.818 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.825+0000 7f2824aae640 4 set_mon_vals no callback set 2024-05-15T12:43:11.818 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.825+0000 7f2816ffd640 10 monclient: dump: 2024-05-15T12:43:11.818 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:11.818 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:11.818 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.818 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:11.818 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:11.818 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:11.818 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:11.818 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:11.818 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:11.818 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:11.818 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.825+0000 7f2828515640 10 monclient: _renew_subs 2024-05-15T12:43:11.819 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.825+0000 7f2828515640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:43:11.819 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.825+0000 7f2828515640 1 -- 172.21.15.117:0/1602088070 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28201d8a10 con 0x7f28200a94b0 2024-05-15T12:43:11.819 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.826+0000 7f2828515640 1 librados: init done 2024-05-15T12:43:11.819 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.826+0000 7f8382ffd640 1 --2- 172.21.15.117:0/568609185 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f83840b4d00 0x7f838413e0f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f83780090b0 tx=0x7f83780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:11.819 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.826+0000 7f836bfff640 10 client.?.objecter ms_handle_connect 0x7f83840b4d00 2024-05-15T12:43:11.819 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.826+0000 7f836bfff640 10 client.?.objecter resend_mon_ops 2024-05-15T12:43:11.819 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.827+0000 7f2828515640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:11.819 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.827+0000 7f2828515640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:11.820 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.827+0000 7f2828515640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:43:11.820 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.827+0000 7f2828515640 1 -- 172.21.15.117:0/1602088070 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f282004f0e0 con 0x7f28200a94b0 2024-05-15T12:43:11.907 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.917+0000 7fb482928640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-05-15T12:43:11.907 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.917+0000 7fb482928640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-05-15T12:43:11.907 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.917+0000 7fb482928640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T12:43:11.907 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.917+0000 7fb482928640 1 -- 172.21.15.117:0/1085814967 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fb47c146750 con 0x7fb47c0a94b0 2024-05-15T12:43:11.946 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.956+0000 7fd9e3a8f640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2024-05-15T12:43:11.946 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.956+0000 7fd9e3a8f640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-05-15T12:43:11.946 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.956+0000 7fd9e3a8f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:43:11.946 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.956+0000 7fd9e3a8f640 1 -- 172.21.15.117:0/1186925958 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fd9dc0acc50 con 0x7fd9dc14ab70 2024-05-15T12:43:11.973 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.983+0000 7f19d67fc640 1 -- 172.21.15.117:0/3224956099 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f19dc058500 con 0x7f19cc0053c0 2024-05-15T12:43:11.973 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.983+0000 7f19d67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:11.973 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.983+0000 7f19d67fc640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:43:11.987 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.997+0000 7f9ee2ffd640 1 -- 172.21.15.117:0/723854573 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9ee807b210 con 0x7f9eec033930 2024-05-15T12:43:11.987 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.997+0000 7f9ee2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-05-15T12:43:11.987 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.997+0000 7f9ee2ffd640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:43:11.988 INFO:teuthology.orchestra.run.smithi117.stdout:85899345930 2024-05-15T12:43:11.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.999+0000 7f9ef359e640 10 librados: watch_flush enter 2024-05-15T12:43:11.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.999+0000 7f9ef359e640 10 librados: watch_flush exit 2024-05-15T12:43:11.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.999+0000 7f9ef359e640 20 client.4348.objecter shutdown clearing up homeless session... 2024-05-15T12:43:11.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.999+0000 7f9ef359e640 10 client.4348.objecter successfully canceled tick 2024-05-15T12:43:11.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.999+0000 7f9ef359e640 1 -- 172.21.15.117:0/723854573 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f9ed406ceb0 msgr2=0x7f9ed406f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.999+0000 7f9ef359e640 1 --2- 172.21.15.117:0/723854573 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f9ed406ceb0 0x7f9ed406f370 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9ee40010e0 tx=0x7f9ee4007be0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.999+0000 7f9ef359e640 10 monclient: shutdown 2024-05-15T12:43:11.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.999+0000 7f9ef359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:11.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.999+0000 7f9ef359e640 1 -- 172.21.15.117:0/723854573 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9eec033930 msgr2=0x7f9eec0c1570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:11.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.999+0000 7f9ef359e640 1 --2- 172.21.15.117:0/723854573 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9eec033930 0x7f9eec0c1570 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9ee80098c0 tx=0x7f9ee80046c0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.999+0000 7f9ef359e640 1 -- 172.21.15.117:0/723854573 shutdown_connections 2024-05-15T12:43:11.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.999+0000 7f9ef359e640 1 --2- 172.21.15.117:0/723854573 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9eec160c30 0x7f9eec0c1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.999+0000 7f9ef359e640 1 --2- 172.21.15.117:0/723854573 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f9ed406ceb0 0x7f9ed406f370 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.999+0000 7f9ef359e640 1 --2- 172.21.15.117:0/723854573 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9eec033930 0x7f9eec0c1570 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.999+0000 7f9ef359e640 1 --2- 172.21.15.117:0/723854573 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9eec006fd0 0x7f9eec160210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:11.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.999+0000 7f9ef359e640 1 -- 172.21.15.117:0/723854573 >> 172.21.15.117:0/723854573 conn(0x7f9eec0061a0 msgr2=0x7f9eec03c440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:11.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:11.999+0000 7f9ef359e640 1 -- 172.21.15.117:0/723854573 shutdown_connections 2024-05-15T12:43:11.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.000+0000 7f9ef359e640 1 -- 172.21.15.117:0/723854573 wait complete. 2024-05-15T12:43:11.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.000+0000 7f9ef359e640 1 librados: shutdown 2024-05-15T12:43:12.001 INFO:tasks.ceph.ceph_manager.ceph:need seq 85899345931 got 85899345930 for osd.0 2024-05-15T12:43:12.016 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.025+0000 7f4e3effd640 1 -- 172.21.15.117:0/1274867206 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e38017070 con 0x7f4e480b1f30 2024-05-15T12:43:12.016 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.025+0000 7f4e3effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:12.016 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.025+0000 7f4e3effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:43:12.016 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.025+0000 7f4e3effd640 10 monclient: dump: 2024-05-15T12:43:12.016 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:12.016 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:12.016 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:12.017 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:12.017 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:12.017 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:12.017 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:12.017 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:12.017 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:12.017 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:12.017 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.025+0000 7f4e3effd640 10 monclient: _finish_auth 0 2024-05-15T12:43:12.017 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.025+0000 7f4e3effd640 10 monclient: _check_auth_tickets 2024-05-15T12:43:12.017 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.025+0000 7f4e3effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:12.017 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.025+0000 7f4e3effd640 1 -- 172.21.15.117:0/1274867206 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e380035f0 con 0x7f4e480b1f30 2024-05-15T12:43:12.017 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.025+0000 7f4e505d1640 5 monclient: authenticate success, global_id 4304 2024-05-15T12:43:12.017 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.025+0000 7f4e505d1640 10 monclient: _renew_subs 2024-05-15T12:43:12.017 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.025+0000 7f4e505d1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T12:43:12.018 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.025+0000 7f4e505d1640 1 -- 172.21.15.117:0/1274867206 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e480af2a0 con 0x7f4e480b1f30 2024-05-15T12:43:12.018 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.026+0000 7f4e3effd640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:12.018 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.026+0000 7f4e3effd640 1 -- 172.21.15.117:0/1274867206 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e38003a20 con 0x7f4e480b1f30 2024-05-15T12:43:12.018 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.026+0000 7f4e3effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:12.018 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.026+0000 7f4e3effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T12:43:12.018 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.026+0000 7f4e3effd640 10 monclient: dump: 2024-05-15T12:43:12.018 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:12.018 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:12.018 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:12.018 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:12.018 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:12.018 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:12.018 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:12.018 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:12.019 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:12.019 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:12.019 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.026+0000 7f4e4cb6a640 4 set_mon_vals no callback set 2024-05-15T12:43:12.019 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.026+0000 7f4e505d1640 10 client.4304.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:43:12.019 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.026+0000 7f4e505d1640 10 monclient: _renew_subs 2024-05-15T12:43:12.019 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.026+0000 7f4e505d1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T12:43:12.019 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.026+0000 7f4e505d1640 1 -- 172.21.15.117:0/1274867206 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e480aa500 con 0x7f4e480b1f30 2024-05-15T12:43:12.019 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.026+0000 7f4e505d1640 1 librados: init done 2024-05-15T12:43:12.019 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.026+0000 7f4e505d1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:12.020 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.026+0000 7f4e505d1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:12.020 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.026+0000 7f4e505d1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T12:43:12.020 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.026+0000 7f4e505d1640 1 -- 172.21.15.117:0/1274867206 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e4804f0e0 con 0x7f4e480b1f30 2024-05-15T12:43:12.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.037+0000 7fede3fff640 1 -- 172.21.15.117:0/1283422087 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fedf804ffd0 con 0x7fedf00c64a0 2024-05-15T12:43:12.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.037+0000 7fede3fff640 10 client.4298.objecter ms_dispatch 0x7fedf8000b90 osd_map(26..26 src has 1..26) v4 2024-05-15T12:43:12.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.037+0000 7fede3fff640 3 client.4298.objecter handle_osd_map got epochs [26,26] > 0 2024-05-15T12:43:12.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.037+0000 7fede3fff640 3 client.4298.objecter handle_osd_map decoding full epoch 26 2024-05-15T12:43:12.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.037+0000 7fede3fff640 20 client.4298.objecter dump_active .. 0 homeless 2024-05-15T12:43:12.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.037+0000 7fede3fff640 1 -- 172.21.15.117:0/1283422087 <== mon.2 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fedf80a2080 con 0x7fedf00c64a0 2024-05-15T12:43:12.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.037+0000 7fede3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:12.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.037+0000 7fede3fff640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:43:12.132 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.142+0000 7f2816ffd640 1 -- 172.21.15.117:0/1602088070 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f281c017070 con 0x7f28200a94b0 2024-05-15T12:43:12.133 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.143+0000 7f2816ffd640 1 --2- 172.21.15.117:0/1602088070 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f280406ce60 0x7f280406f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:12.133 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.143+0000 7f2825ab0640 1 --2- 172.21.15.117:0/1602088070 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f280406ce60 0x7f280406f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:12.133 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.143+0000 7f2825ab0640 10 monclient: get_auth_request con 0x7f280406ce60 auth_method 0 2024-05-15T12:43:12.134 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.144+0000 7f2816ffd640 10 client.4335.objecter ms_handle_connect 0x7f280406ce60 2024-05-15T12:43:12.134 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.144+0000 7f2825ab0640 1 --2- 172.21.15.117:0/1602088070 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f280406ce60 0x7f280406f320 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f28200673e0 tx=0x7f2810007a40 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:12.164 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.174+0000 7f2816ffd640 1 -- 172.21.15.117:0/1602088070 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f281c056910 con 0x7f28200a94b0 2024-05-15T12:43:12.164 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.174+0000 7f2816ffd640 10 client.4335.objecter ms_dispatch 0x7f2820000b90 osd_map(26..26 src has 1..26) v4 2024-05-15T12:43:12.164 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.174+0000 7f2816ffd640 3 client.4335.objecter handle_osd_map got epochs [26,26] > 0 2024-05-15T12:43:12.164 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.174+0000 7f2816ffd640 3 client.4335.objecter handle_osd_map decoding full epoch 26 2024-05-15T12:43:12.164 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.174+0000 7f2816ffd640 20 client.4335.objecter dump_active .. 0 homeless 2024-05-15T12:43:12.164 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.174+0000 7f2816ffd640 1 -- 172.21.15.117:0/1602088070 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f281c058f70 con 0x7f28200a94b0 2024-05-15T12:43:12.164 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.174+0000 7f2816ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:12.164 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.174+0000 7f2816ffd640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:43:12.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.180+0000 7fd9d27fc640 1 -- 172.21.15.117:0/1186925958 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd9d8040070 con 0x7fd9dc14ab70 2024-05-15T12:43:12.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.180+0000 7fd9d27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-05-15T12:43:12.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.180+0000 7fd9d27fc640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:43:12.170 INFO:teuthology.orchestra.run.smithi117.stdout:55834574854 2024-05-15T12:43:12.172 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.182+0000 7fd9e3a8f640 10 librados: watch_flush enter 2024-05-15T12:43:12.173 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.182+0000 7fd9e3a8f640 10 librados: watch_flush exit 2024-05-15T12:43:12.173 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.182+0000 7fd9e3a8f640 20 client.4329.objecter shutdown clearing up homeless session... 2024-05-15T12:43:12.173 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.182+0000 7fd9e3a8f640 10 client.4329.objecter successfully canceled tick 2024-05-15T12:43:12.173 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.182+0000 7fd9e3a8f640 1 -- 172.21.15.117:0/1186925958 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fd9c006ce60 msgr2=0x7fd9c006f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:12.173 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.182+0000 7fd9e3a8f640 1 --2- 172.21.15.117:0/1186925958 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fd9c006ce60 0x7fd9c006f320 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc0011e0 tx=0x7fd9cc007890 comp rx=0 tx=0).stop 2024-05-15T12:43:12.173 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.182+0000 7fd9e3a8f640 10 monclient: shutdown 2024-05-15T12:43:12.173 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.182+0000 7fd9e3a8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:12.173 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.182+0000 7fd9e3a8f640 1 -- 172.21.15.117:0/1186925958 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9dc14ab70 msgr2=0x7fd9dc144d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:12.173 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.182+0000 7fd9e3a8f640 1 --2- 172.21.15.117:0/1186925958 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9dc14ab70 0x7fd9dc144d80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd9d8009890 tx=0x7fd9d80046c0 comp rx=0 tx=0).stop 2024-05-15T12:43:12.173 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.182+0000 7fd9e3a8f640 1 -- 172.21.15.117:0/1186925958 shutdown_connections 2024-05-15T12:43:12.173 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.182+0000 7fd9e3a8f640 1 --2- 172.21.15.117:0/1186925958 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9dc14ab70 0x7fd9dc144d80 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:12.173 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.182+0000 7fd9e3a8f640 1 --2- 172.21.15.117:0/1186925958 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fd9c006ce60 0x7fd9c006f320 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:12.173 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.182+0000 7fd9e3a8f640 1 --2- 172.21.15.117:0/1186925958 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd9dc06eac0 0x7fd9dc144840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:12.173 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.182+0000 7fd9e3a8f640 1 --2- 172.21.15.117:0/1186925958 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd9dc06bd70 0x7fd9dc0ab630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:12.174 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.182+0000 7fd9e3a8f640 1 -- 172.21.15.117:0/1186925958 >> 172.21.15.117:0/1186925958 conn(0x7fd9dc06ada0 msgr2=0x7fd9dc06b180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:12.174 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.182+0000 7fd9e3a8f640 1 -- 172.21.15.117:0/1186925958 shutdown_connections 2024-05-15T12:43:12.174 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.182+0000 7fd9e3a8f640 1 -- 172.21.15.117:0/1186925958 wait complete. 2024-05-15T12:43:12.174 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.182+0000 7fd9e3a8f640 1 librados: shutdown 2024-05-15T12:43:12.183 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.7 2024-05-15T12:43:12.187 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.198+0000 7f19e6820640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-05-15T12:43:12.187 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.198+0000 7f19e6820640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-05-15T12:43:12.187 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.198+0000 7f19e6820640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:43:12.187 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.198+0000 7f19e6820640 1 -- 172.21.15.117:0/3224956099 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f19d80057e0 con 0x7f19cc0053c0 2024-05-15T12:43:12.191 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.202+0000 7f19d67fc640 1 -- 172.21.15.117:0/3224956099 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f19dc057e00 con 0x7f19cc0053c0 2024-05-15T12:43:12.191 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.202+0000 7f19d67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-05-15T12:43:12.191 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.202+0000 7f19d67fc640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:43:12.191 INFO:teuthology.orchestra.run.smithi117.stdout:85899345928 2024-05-15T12:43:12.193 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.203+0000 7f19e6820640 10 librados: watch_flush enter 2024-05-15T12:43:12.193 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.203+0000 7f19e6820640 10 librados: watch_flush exit 2024-05-15T12:43:12.193 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.203+0000 7f19e6820640 20 client.4351.objecter shutdown clearing up homeless session... 2024-05-15T12:43:12.193 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.203+0000 7f19e6820640 10 client.4351.objecter successfully canceled tick 2024-05-15T12:43:12.193 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.204+0000 7f19e6820640 1 -- 172.21.15.117:0/3224956099 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f19c406ce60 msgr2=0x7f19c406f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:12.194 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.204+0000 7f19e6820640 1 --2- 172.21.15.117:0/3224956099 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f19c406ce60 0x7f19c406f320 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f19d0000f60 tx=0x7f19d00014f0 comp rx=0 tx=0).stop 2024-05-15T12:43:12.194 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.204+0000 7f19e6820640 10 monclient: shutdown 2024-05-15T12:43:12.194 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.204+0000 7f19e6820640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:12.194 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.204+0000 7f19e6820640 1 -- 172.21.15.117:0/3224956099 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f19cc0053c0 msgr2=0x7f19d80b59e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:12.194 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.204+0000 7f19e6820640 1 --2- 172.21.15.117:0/3224956099 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f19cc0053c0 0x7f19d80b59e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f19dc00c440 tx=0x7f19dc00c900 comp rx=0 tx=0).stop 2024-05-15T12:43:12.194 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.204+0000 7f19e6820640 1 -- 172.21.15.117:0/3224956099 shutdown_connections 2024-05-15T12:43:12.194 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.204+0000 7f19e6820640 1 --2- 172.21.15.117:0/3224956099 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f19d802fff0 0x7f19d80b2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:12.194 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.204+0000 7f19e6820640 1 --2- 172.21.15.117:0/3224956099 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f19d802d8c0 0x7f19d802f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:12.194 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.204+0000 7f19e6820640 1 --2- 172.21.15.117:0/3224956099 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f19c406ce60 0x7f19c406f320 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:12.194 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.204+0000 7f19e6820640 1 --2- 172.21.15.117:0/3224956099 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f19cc0053c0 0x7f19d80b59e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:12.194 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.204+0000 7f19e6820640 1 -- 172.21.15.117:0/3224956099 >> 172.21.15.117:0/3224956099 conn(0x7f19d8006ab0 msgr2=0x7f19d80c4a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:12.194 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.204+0000 7f19e6820640 1 -- 172.21.15.117:0/3224956099 shutdown_connections 2024-05-15T12:43:12.194 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.204+0000 7f19e6820640 1 -- 172.21.15.117:0/3224956099 wait complete. 2024-05-15T12:43:12.194 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.204+0000 7f19e6820640 1 librados: shutdown 2024-05-15T12:43:12.209 INFO:tasks.ceph.ceph_manager.ceph:need seq 85899345929 got 85899345928 for osd.1 2024-05-15T12:43:12.239 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.249+0000 7f836bfff640 1 -- 172.21.15.117:0/568609185 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8378032860 con 0x7f83840b4d00 2024-05-15T12:43:12.239 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.249+0000 7f836bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:12.239 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.249+0000 7f836bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T12:43:12.239 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.249+0000 7f836bfff640 10 monclient: dump: 2024-05-15T12:43:12.239 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:12.239 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:12.239 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:12.240 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:12.240 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:12.240 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:12.240 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:12.240 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:12.240 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:12.240 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:12.240 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.249+0000 7f836bfff640 10 monclient: _finish_auth 0 2024-05-15T12:43:12.240 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.249+0000 7f836bfff640 10 monclient: _check_auth_tickets 2024-05-15T12:43:12.240 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.249+0000 7f836bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:12.240 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.249+0000 7f836bfff640 1 -- 172.21.15.117:0/568609185 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8378032a00 con 0x7f83840b4d00 2024-05-15T12:43:12.240 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.249+0000 7f836bfff640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:12.240 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.249+0000 7f836bfff640 1 -- 172.21.15.117:0/568609185 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8378032df0 con 0x7f83840b4d00 2024-05-15T12:43:12.240 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.249+0000 7f836bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:12.240 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.249+0000 7f836bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T12:43:12.241 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.249+0000 7f836bfff640 10 monclient: dump: 2024-05-15T12:43:12.241 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:12.241 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:12.241 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:12.241 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:12.241 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:12.241 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:12.241 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:12.241 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:12.241 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:12.241 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:12.241 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.249+0000 7f8381ffb640 4 set_mon_vals no callback set 2024-05-15T12:43:12.241 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.250+0000 7f8389c6a640 5 monclient: authenticate success, global_id 4338 2024-05-15T12:43:12.241 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.250+0000 7f8389c6a640 10 monclient: _renew_subs 2024-05-15T12:43:12.241 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.250+0000 7f8389c6a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:43:12.242 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.250+0000 7f8389c6a640 1 -- 172.21.15.117:0/568609185 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83840ac4a0 con 0x7f83840b4d00 2024-05-15T12:43:12.242 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.250+0000 7f8389c6a640 10 client.4338.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:43:12.242 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.250+0000 7f8389c6a640 10 monclient: _renew_subs 2024-05-15T12:43:12.242 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.250+0000 7f8389c6a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:43:12.242 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.250+0000 7f8389c6a640 1 -- 172.21.15.117:0/568609185 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f838413f410 con 0x7f83840b4d00 2024-05-15T12:43:12.242 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.250+0000 7f8389c6a640 1 librados: init done 2024-05-15T12:43:12.242 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.251+0000 7f8389c6a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:12.243 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.251+0000 7f8389c6a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:12.243 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.251+0000 7f8389c6a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:43:12.243 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.251+0000 7f8389c6a640 1 -- 172.21.15.117:0/568609185 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83840b5b20 con 0x7f83840b4d00 2024-05-15T12:43:12.244 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.255+0000 7fedfc8ba640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-05-15T12:43:12.245 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.256+0000 7fedfc8ba640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-05-15T12:43:12.245 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.256+0000 7fedfc8ba640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T12:43:12.245 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.256+0000 7fedfc8ba640 1 -- 172.21.15.117:0/1283422087 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fedf00c74a0 con 0x7fedf00c64a0 2024-05-15T12:43:12.315 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.325+0000 7fb478ff9640 1 -- 172.21.15.117:0/1085814967 <== mon.2 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb46c058970 con 0x7fb47c0a94b0 2024-05-15T12:43:12.315 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.326+0000 7fb478ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-05-15T12:43:12.315 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.326+0000 7fb478ff9640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:43:12.315 INFO:teuthology.orchestra.run.smithi117.stdout:85899345929 2024-05-15T12:43:12.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.327+0000 7fb482928640 10 librados: watch_flush enter 2024-05-15T12:43:12.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.328+0000 7fb482928640 10 librados: watch_flush exit 2024-05-15T12:43:12.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.328+0000 7fb482928640 20 client.4301.objecter shutdown clearing up homeless session... 2024-05-15T12:43:12.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.328+0000 7fb482928640 10 client.4301.objecter successfully canceled tick 2024-05-15T12:43:12.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.328+0000 7fb482928640 1 -- 172.21.15.117:0/1085814967 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fb46806ceb0 msgr2=0x7fb46806f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:12.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.328+0000 7fb482928640 1 --2- 172.21.15.117:0/1085814967 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fb46806ceb0 0x7fb46806f370 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb47c0aed50 tx=0x7fb470008040 comp rx=0 tx=0).stop 2024-05-15T12:43:12.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.328+0000 7fb482928640 10 monclient: shutdown 2024-05-15T12:43:12.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.328+0000 7fb482928640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:12.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.328+0000 7fb482928640 1 -- 172.21.15.117:0/1085814967 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb47c0a94b0 msgr2=0x7fb47c1418d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:12.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.328+0000 7fb482928640 1 --2- 172.21.15.117:0/1085814967 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb47c0a94b0 0x7fb47c1418d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb46c00c410 tx=0x7fb46c00c8d0 comp rx=0 tx=0).stop 2024-05-15T12:43:12.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.328+0000 7fb482928640 1 -- 172.21.15.117:0/1085814967 shutdown_connections 2024-05-15T12:43:12.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.328+0000 7fb482928640 1 --2- 172.21.15.117:0/1085814967 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fb46806ceb0 0x7fb46806f370 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:12.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.328+0000 7fb482928640 1 --2- 172.21.15.117:0/1085814967 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb47c13a1a0 0x7fb47c13a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:12.319 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.328+0000 7fb482928640 1 --2- 172.21.15.117:0/1085814967 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb47c0b4d00 0x7fb47c1d8c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:12.319 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.328+0000 7fb482928640 1 --2- 172.21.15.117:0/1085814967 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb47c0a94b0 0x7fb47c1418d0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:12.319 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.328+0000 7fb482928640 1 -- 172.21.15.117:0/1085814967 >> 172.21.15.117:0/1085814967 conn(0x7fb47c066cc0 msgr2=0x7fb47c0b86c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:12.319 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.328+0000 7fb482928640 1 -- 172.21.15.117:0/1085814967 shutdown_connections 2024-05-15T12:43:12.319 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.328+0000 7fb482928640 1 -- 172.21.15.117:0/1085814967 wait complete. 2024-05-15T12:43:12.319 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.328+0000 7fb482928640 1 librados: shutdown 2024-05-15T12:43:12.332 INFO:tasks.ceph.ceph_manager.ceph:need seq 85899345930 got 85899345929 for osd.2 2024-05-15T12:43:12.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.344+0000 7f4e3effd640 1 -- 172.21.15.117:0/1274867206 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f4e3801a070 con 0x7f4e480b1f30 2024-05-15T12:43:12.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.345+0000 7f4e3effd640 1 --2- 172.21.15.117:0/1274867206 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f4e2c06ceb0 0x7f4e2c06f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:12.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.345+0000 7f4e4db6c640 1 --2- 172.21.15.117:0/1274867206 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f4e2c06ceb0 0x7f4e2c06f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:12.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.345+0000 7f4e4db6c640 10 monclient: get_auth_request con 0x7f4e2c06ceb0 auth_method 0 2024-05-15T12:43:12.335 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.346+0000 7f4e4db6c640 1 --2- 172.21.15.117:0/1274867206 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f4e2c06ceb0 0x7f4e2c06f370 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4e480b0900 tx=0x7f4e44009060 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:12.335 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.346+0000 7f4e3effd640 10 client.4304.objecter ms_handle_connect 0x7f4e2c06ceb0 2024-05-15T12:43:12.355 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.366+0000 7f4e3effd640 1 -- 172.21.15.117:0/1274867206 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f4e38056310 con 0x7f4e480b1f30 2024-05-15T12:43:12.355 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.366+0000 7f4e3effd640 10 client.4304.objecter ms_dispatch 0x7f4e48000b90 osd_map(26..26 src has 1..26) v4 2024-05-15T12:43:12.355 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.366+0000 7f4e3effd640 3 client.4304.objecter handle_osd_map got epochs [26,26] > 0 2024-05-15T12:43:12.355 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.366+0000 7f4e3effd640 3 client.4304.objecter handle_osd_map decoding full epoch 26 2024-05-15T12:43:12.355 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.366+0000 7f4e3effd640 20 client.4304.objecter dump_active .. 0 homeless 2024-05-15T12:43:12.355 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.366+0000 7f4e3effd640 1 -- 172.21.15.117:0/1274867206 <== mon.2 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f4e38021020 con 0x7f4e480b1f30 2024-05-15T12:43:12.355 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.366+0000 7f4e3effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:12.355 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.366+0000 7f4e3effd640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:43:12.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.372+0000 7fede3fff640 1 -- 172.21.15.117:0/1283422087 <== mon.2 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fedf808f030 con 0x7fedf00c64a0 2024-05-15T12:43:12.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.372+0000 7fede3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-05-15T12:43:12.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.372+0000 7fede3fff640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:43:12.362 INFO:teuthology.orchestra.run.smithi117.stdout:81604378632 2024-05-15T12:43:12.364 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.375+0000 7fedfc8ba640 10 librados: watch_flush enter 2024-05-15T12:43:12.364 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.375+0000 7fedfc8ba640 10 librados: watch_flush exit 2024-05-15T12:43:12.364 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.375+0000 7fedfc8ba640 20 client.4298.objecter shutdown clearing up homeless session... 2024-05-15T12:43:12.364 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.375+0000 7fedfc8ba640 10 client.4298.objecter successfully canceled tick 2024-05-15T12:43:12.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.375+0000 7fedfc8ba640 1 -- 172.21.15.117:0/1283422087 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7feddc06ceb0 msgr2=0x7feddc06f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:12.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.375+0000 7fedfc8ba640 1 --2- 172.21.15.117:0/1283422087 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7feddc06ceb0 0x7feddc06f370 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fedec034000 tx=0x7fedec008030 comp rx=0 tx=0).stop 2024-05-15T12:43:12.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.375+0000 7fedfc8ba640 10 monclient: shutdown 2024-05-15T12:43:12.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.375+0000 7fedfc8ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:12.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.375+0000 7fedfc8ba640 1 -- 172.21.15.117:0/1283422087 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fedf00c64a0 msgr2=0x7fedf0152070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:12.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.375+0000 7fedfc8ba640 1 --2- 172.21.15.117:0/1283422087 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fedf00c64a0 0x7fedf0152070 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fedf80508c0 tx=0x7fedf806a0a0 comp rx=0 tx=0).stop 2024-05-15T12:43:12.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.375+0000 7fedfc8ba640 1 -- 172.21.15.117:0/1283422087 shutdown_connections 2024-05-15T12:43:12.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.375+0000 7fedfc8ba640 1 --2- 172.21.15.117:0/1283422087 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fedf00c64a0 0x7fedf0152070 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:12.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.375+0000 7fedfc8ba640 1 --2- 172.21.15.117:0/1283422087 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7feddc06ceb0 0x7feddc06f370 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:12.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.375+0000 7fedfc8ba640 1 --2- 172.21.15.117:0/1283422087 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fedf0033930 0x7fedf0151b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:12.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.375+0000 7fedfc8ba640 1 --2- 172.21.15.117:0/1283422087 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fedf0006fd0 0x7fedf0154710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:12.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.375+0000 7fedfc8ba640 1 -- 172.21.15.117:0/1283422087 >> 172.21.15.117:0/1283422087 conn(0x7fedf00061a0 msgr2=0x7fedf00cb7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:12.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.375+0000 7fedfc8ba640 1 -- 172.21.15.117:0/1283422087 shutdown_connections 2024-05-15T12:43:12.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.375+0000 7fedfc8ba640 1 -- 172.21.15.117:0/1283422087 wait complete. 2024-05-15T12:43:12.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.375+0000 7fedfc8ba640 1 librados: shutdown 2024-05-15T12:43:12.376 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378634 got 81604378632 for osd.4 2024-05-15T12:43:12.376 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.387+0000 7f2828515640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-05-15T12:43:12.377 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.387+0000 7f2828515640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-05-15T12:43:12.377 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.387+0000 7f2828515640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:43:12.377 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.387+0000 7f2828515640 1 -- 172.21.15.117:0/1602088070 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f28200b7260 con 0x7f28200a94b0 2024-05-15T12:43:12.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.521+0000 7f836bfff640 1 -- 172.21.15.117:0/568609185 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f837804c060 con 0x7f83840b4d00 2024-05-15T12:43:12.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.521+0000 7f836bfff640 1 --2- 172.21.15.117:0/568609185 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f837006ceb0 0x7f837006f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:12.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.522+0000 7f83837fe640 1 --2- 172.21.15.117:0/568609185 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f837006ceb0 0x7f837006f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:12.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.522+0000 7f83837fe640 10 monclient: get_auth_request con 0x7f837006ceb0 auth_method 0 2024-05-15T12:43:12.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.522+0000 7f836bfff640 10 client.4338.objecter ms_handle_connect 0x7f837006ceb0 2024-05-15T12:43:12.512 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.522+0000 7f83837fe640 1 --2- 172.21.15.117:0/568609185 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f837006ceb0 0x7f837006f370 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8374001560 tx=0x7f8374007990 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:12.525 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.536+0000 7f836bfff640 1 -- 172.21.15.117:0/568609185 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f8378078a00 con 0x7f83840b4d00 2024-05-15T12:43:12.525 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.536+0000 7f836bfff640 10 client.4338.objecter ms_dispatch 0x7f8384000b90 osd_map(26..26 src has 1..26) v4 2024-05-15T12:43:12.525 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.536+0000 7f836bfff640 3 client.4338.objecter handle_osd_map got epochs [26,26] > 0 2024-05-15T12:43:12.525 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.536+0000 7f836bfff640 3 client.4338.objecter handle_osd_map decoding full epoch 26 2024-05-15T12:43:12.525 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.536+0000 7f836bfff640 20 client.4338.objecter dump_active .. 0 homeless 2024-05-15T12:43:12.526 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.536+0000 7f836bfff640 1 -- 172.21.15.117:0/568609185 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f83840ac4a0 con 0x7f83840b4d00 2024-05-15T12:43:12.526 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.536+0000 7f836bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:12.526 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.536+0000 7f836bfff640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:43:12.538 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.549+0000 7f2816ffd640 1 -- 172.21.15.117:0/1602088070 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f281c058870 con 0x7f28200a94b0 2024-05-15T12:43:12.538 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.549+0000 7f2816ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-05-15T12:43:12.538 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.549+0000 7f2816ffd640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:43:12.538 INFO:teuthology.orchestra.run.smithi117.stdout:60129542149 2024-05-15T12:43:12.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.552+0000 7f2828515640 10 librados: watch_flush enter 2024-05-15T12:43:12.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.552+0000 7f2828515640 10 librados: watch_flush exit 2024-05-15T12:43:12.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.552+0000 7f2828515640 20 client.4335.objecter shutdown clearing up homeless session... 2024-05-15T12:43:12.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.552+0000 7f2828515640 10 client.4335.objecter successfully canceled tick 2024-05-15T12:43:12.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.552+0000 7f2828515640 1 -- 172.21.15.117:0/1602088070 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f280406ce60 msgr2=0x7f280406f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:12.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.552+0000 7f2828515640 1 --2- 172.21.15.117:0/1602088070 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f280406ce60 0x7f280406f320 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f28200673e0 tx=0x7f2810007a40 comp rx=0 tx=0).stop 2024-05-15T12:43:12.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.552+0000 7f2828515640 10 monclient: shutdown 2024-05-15T12:43:12.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.552+0000 7f2828515640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:12.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.552+0000 7f2828515640 1 -- 172.21.15.117:0/1602088070 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f28200a94b0 msgr2=0x7f282013e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:12.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.552+0000 7f2828515640 1 --2- 172.21.15.117:0/1602088070 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f28200a94b0 0x7f282013e810 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f281c00b4a0 tx=0x7f281c00b960 comp rx=0 tx=0).stop 2024-05-15T12:43:12.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.552+0000 7f2828515640 1 -- 172.21.15.117:0/1602088070 shutdown_connections 2024-05-15T12:43:12.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.552+0000 7f2828515640 1 --2- 172.21.15.117:0/1602088070 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f280406ce60 0x7f280406f320 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:12.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.552+0000 7f2828515640 1 --2- 172.21.15.117:0/1602088070 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f28201d8380 0x7f2820142c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:12.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.552+0000 7f2828515640 1 --2- 172.21.15.117:0/1602088070 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f28200b4d00 0x7f2820142580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:12.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.552+0000 7f2828515640 1 --2- 172.21.15.117:0/1602088070 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f28200a94b0 0x7f282013e810 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:12.543 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.552+0000 7f2828515640 1 -- 172.21.15.117:0/1602088070 >> 172.21.15.117:0/1602088070 conn(0x7f2820066cc0 msgr2=0x7f28200b86c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:12.543 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.552+0000 7f2828515640 1 -- 172.21.15.117:0/1602088070 shutdown_connections 2024-05-15T12:43:12.543 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.552+0000 7f2828515640 1 -- 172.21.15.117:0/1602088070 wait complete. 2024-05-15T12:43:12.543 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.552+0000 7f2828515640 1 librados: shutdown 2024-05-15T12:43:12.554 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.564+0000 7f4e505d1640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2024-05-15T12:43:12.554 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.564+0000 7f4e505d1640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-05-15T12:43:12.554 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.564+0000 7f4e505d1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T12:43:12.554 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.564+0000 7f4e505d1640 1 -- 172.21.15.117:0/1274867206 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f4e48136120 con 0x7f4e480b1f30 2024-05-15T12:43:12.556 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542149 for osd.5 2024-05-15T12:43:12.613 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.624+0000 7f4e3effd640 1 -- 172.21.15.117:0/1274867206 <== mon.2 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4e38058270 con 0x7f4e480b1f30 2024-05-15T12:43:12.614 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.624+0000 7f4e3effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-05-15T12:43:12.614 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.624+0000 7f4e3effd640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:43:12.614 INFO:teuthology.orchestra.run.smithi117.stdout:81604378630 2024-05-15T12:43:12.619 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.627+0000 7f4e505d1640 10 librados: watch_flush enter 2024-05-15T12:43:12.619 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.627+0000 7f4e505d1640 10 librados: watch_flush exit 2024-05-15T12:43:12.619 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.627+0000 7f4e505d1640 20 client.4304.objecter shutdown clearing up homeless session... 2024-05-15T12:43:12.619 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.627+0000 7f4e505d1640 10 client.4304.objecter successfully canceled tick 2024-05-15T12:43:12.619 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.627+0000 7f4e505d1640 1 -- 172.21.15.117:0/1274867206 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f4e2c06ceb0 msgr2=0x7f4e2c06f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:12.619 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.627+0000 7f4e505d1640 1 --2- 172.21.15.117:0/1274867206 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f4e2c06ceb0 0x7f4e2c06f370 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4e480b0900 tx=0x7f4e44009060 comp rx=0 tx=0).stop 2024-05-15T12:43:12.619 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.627+0000 7f4e505d1640 10 monclient: shutdown 2024-05-15T12:43:12.619 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.627+0000 7f4e505d1640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:12.619 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.627+0000 7f4e505d1640 1 -- 172.21.15.117:0/1274867206 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4e480b1f30 msgr2=0x7f4e480afa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:12.619 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.627+0000 7f4e505d1640 1 --2- 172.21.15.117:0/1274867206 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4e480b1f30 0x7f4e480afa30 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4e3800c410 tx=0x7f4e3800c8d0 comp rx=0 tx=0).stop 2024-05-15T12:43:12.620 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.627+0000 7f4e505d1640 1 -- 172.21.15.117:0/1274867206 shutdown_connections 2024-05-15T12:43:12.620 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.627+0000 7f4e505d1640 1 --2- 172.21.15.117:0/1274867206 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f4e2c06ceb0 0x7f4e2c06f370 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:12.620 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.627+0000 7f4e505d1640 1 --2- 172.21.15.117:0/1274867206 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4e48144500 0x7f4e48130e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:12.620 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.627+0000 7f4e505d1640 1 --2- 172.21.15.117:0/1274867206 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e480b3000 0x7f4e481308e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:12.620 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.627+0000 7f4e505d1640 1 --2- 172.21.15.117:0/1274867206 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4e480b1f30 0x7f4e480afa30 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:12.620 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.627+0000 7f4e505d1640 1 -- 172.21.15.117:0/1274867206 >> 172.21.15.117:0/1274867206 conn(0x7f4e48066cc0 msgr2=0x7f4e48135250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:12.620 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.627+0000 7f4e505d1640 1 -- 172.21.15.117:0/1274867206 shutdown_connections 2024-05-15T12:43:12.620 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.627+0000 7f4e505d1640 1 -- 172.21.15.117:0/1274867206 wait complete. 2024-05-15T12:43:12.620 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.627+0000 7f4e505d1640 1 librados: shutdown 2024-05-15T12:43:12.630 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378632 got 81604378630 for osd.6 2024-05-15T12:43:12.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.725+0000 7f8389c6a640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-05-15T12:43:12.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.725+0000 7f8389c6a640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-05-15T12:43:12.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.725+0000 7f8389c6a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:43:12.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.725+0000 7f8389c6a640 1 -- 172.21.15.117:0/568609185 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f838404f0e0 con 0x7f83840b4d00 2024-05-15T12:43:12.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.729+0000 7f836bfff640 1 -- 172.21.15.117:0/568609185 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8378040070 con 0x7f83840b4d00 2024-05-15T12:43:12.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.729+0000 7f836bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-05-15T12:43:12.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.729+0000 7f836bfff640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:43:12.719 INFO:teuthology.orchestra.run.smithi117.stdout:64424509447 2024-05-15T12:43:12.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.731+0000 7f8389c6a640 10 librados: watch_flush enter 2024-05-15T12:43:12.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.731+0000 7f8389c6a640 10 librados: watch_flush exit 2024-05-15T12:43:12.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.731+0000 7f8389c6a640 20 client.4338.objecter shutdown clearing up homeless session... 2024-05-15T12:43:12.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.731+0000 7f8389c6a640 10 client.4338.objecter successfully canceled tick 2024-05-15T12:43:12.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.731+0000 7f8389c6a640 1 -- 172.21.15.117:0/568609185 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f837006ceb0 msgr2=0x7f837006f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:12.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.732+0000 7f8389c6a640 1 --2- 172.21.15.117:0/568609185 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f837006ceb0 0x7f837006f370 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8374001560 tx=0x7f8374007990 comp rx=0 tx=0).stop 2024-05-15T12:43:12.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.732+0000 7f8389c6a640 10 monclient: shutdown 2024-05-15T12:43:12.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.732+0000 7f8389c6a640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:12.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.732+0000 7f8389c6a640 1 -- 172.21.15.117:0/568609185 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f83840b4d00 msgr2=0x7f838413e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:12.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.732+0000 7f8389c6a640 1 --2- 172.21.15.117:0/568609185 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f83840b4d00 0x7f838413e0f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f83780090b0 tx=0x7f83780046c0 comp rx=0 tx=0).stop 2024-05-15T12:43:12.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.732+0000 7f8389c6a640 1 -- 172.21.15.117:0/568609185 shutdown_connections 2024-05-15T12:43:12.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.732+0000 7f8389c6a640 1 --2- 172.21.15.117:0/568609185 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f83841db630 0x7f838413eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:12.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.732+0000 7f8389c6a640 1 --2- 172.21.15.117:0/568609185 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f83840b4d00 0x7f838413e0f0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:12.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.732+0000 7f8389c6a640 1 --2- 172.21.15.117:0/568609185 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f837006ceb0 0x7f837006f370 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:12.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.732+0000 7f8389c6a640 1 --2- 172.21.15.117:0/568609185 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f83840a94b0 0x7f8384142f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:12.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.732+0000 7f8389c6a640 1 -- 172.21.15.117:0/568609185 >> 172.21.15.117:0/568609185 conn(0x7f8384066cc0 msgr2=0x7f83840b5d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:12.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.732+0000 7f8389c6a640 1 -- 172.21.15.117:0/568609185 shutdown_connections 2024-05-15T12:43:12.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.732+0000 7f8389c6a640 1 -- 172.21.15.117:0/568609185 wait complete. 2024-05-15T12:43:12.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:12.732+0000 7f8389c6a640 1 librados: shutdown 2024-05-15T12:43:12.732 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509447 for osd.3 2024-05-15T12:43:12.732 DEBUG:teuthology.parallel:result is None 2024-05-15T12:43:13.002 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-05-15T12:43:13.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.118+0000 7f8a7c5aa640 10 monclient: get_monmap_and_config 2024-05-15T12:43:13.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.118+0000 7f8a7c5aa640 10 monclient: build_initial_monmap 2024-05-15T12:43:13.108 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.118+0000 7f8a7c5aa640 10 monclient: monmap: 2024-05-15T12:43:13.108 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:13.108 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:13.108 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:43:13.118813+0000 2024-05-15T12:43:13.108 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:43:13.118813+0000 2024-05-15T12:43:13.108 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:13.108 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:13.108 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T12:43:13.108 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T12:43:13.108 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T12:43:13.108 INFO:teuthology.orchestra.run.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T12:43:13.108 INFO:teuthology.orchestra.run.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T12:43:13.108 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:13.110 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.119+0000 7f8a7c5aa640 1 Processor -- start 2024-05-15T12:43:13.110 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.119+0000 7f8a7c5aa640 1 -- start start 2024-05-15T12:43:13.110 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.119+0000 7f8a7c5aa640 10 monclient: init 2024-05-15T12:43:13.111 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.120+0000 7f8a7c5aa640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:13.111 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.120+0000 7f8a7c5aa640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-05-15T12:43:13.111 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.120+0000 7f8a7c5aa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8a7406c900 addr v1:172.21.15.117:6789/0 2024-05-15T12:43:13.111 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.120+0000 7f8a7c5aa640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8a740af1c0 0x7f8a74142c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:13.111 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.120+0000 7f8a7c5aa640 10 monclient(hunting): picked mon.noname-b con 0x7f8a740af1c0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:43:13.111 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.120+0000 7f8a7c5aa640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f8a741431b0 0x7f8a74143660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:13.111 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.120+0000 7f8a7c5aa640 10 monclient(hunting): picked mon.noname-a con 0x7f8a741431b0 addr v2:172.21.15.117:3300/0 2024-05-15T12:43:13.111 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.120+0000 7f8a7c5aa640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a740ae0d0 con 0x7f8a7406c900 2024-05-15T12:43:13.111 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.120+0000 7f8a7c5aa640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:13.111 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.120+0000 7f8a7c5aa640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a7406cce0 con 0x7f8a741431b0 2024-05-15T12:43:13.111 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.120+0000 7f8a7c5aa640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:13.111 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.120+0000 7f8a7c5aa640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a740b0500 con 0x7f8a740af1c0 2024-05-15T12:43:13.111 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.120+0000 7f8a7c5aa640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:13.111 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.120+0000 7f8a7c5aa640 10 monclient(hunting): authenticate will time out at 1356.361076s 2024-05-15T12:43:13.112 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.120+0000 7f8a7ab47640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f8a741431b0 0x7f8a74143660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:13.112 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.120+0000 7f8a79b45640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8a740af1c0 0x7f8a74142c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:13.112 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.121+0000 7f8a79b45640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8a740af1c0 0x7f8a74142c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.117:49054/0 (socket says 172.21.15.117:49054) 2024-05-15T12:43:13.112 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.121+0000 7f8a79b45640 1 -- 172.21.15.117:0/1543042612 learned_addr learned my addr 172.21.15.117:0/1543042612 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:43:13.112 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.121+0000 7f8a79b45640 10 monclient(hunting): get_auth_request con 0x7f8a740af1c0 auth_method 0 2024-05-15T12:43:13.112 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.121+0000 7f8a79b45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:13.112 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.121+0000 7f8a79b45640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:13.112 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.121+0000 7f8a79b45640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:13.112 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.121+0000 7f8a7ab47640 10 monclient(hunting): get_auth_request con 0x7f8a741431b0 auth_method 0 2024-05-15T12:43:13.112 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.121+0000 7f8a7ab47640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:13.112 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.121+0000 7f8a7ab47640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:13.112 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.121+0000 7f8a7ab47640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:13.112 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.121+0000 7f8a79b45640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:13.112 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.121+0000 7f8a79b45640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:13.112 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.121+0000 7f8a79b45640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:13.113 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.122+0000 7f8a7ab47640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:13.113 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.122+0000 7f8a7ab47640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:13.113 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.122+0000 7f8a7ab47640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:13.113 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.123+0000 7f8a79b45640 10 monclient(hunting): handle_auth_done global_id 4307 payload 274 2024-05-15T12:43:13.114 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.123+0000 7f8a79b45640 1 -- 172.21.15.117:0/1543042612 >> v2:172.21.15.117:3300/0 conn(0x7f8a741431b0 msgr2=0x7f8a74143660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:43:13.114 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.123+0000 7f8a79b45640 1 --2- 172.21.15.117:0/1543042612 >> v2:172.21.15.117:3300/0 conn(0x7f8a741431b0 0x7f8a74143660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:13.114 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.123+0000 7f8a79b45640 1 -- 172.21.15.117:0/1543042612 >> v1:172.21.15.117:6789/0 conn(0x7f8a7406c900 legacy=0x7f8a740aeab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:13.114 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.123+0000 7f8a79b45640 10 monclient: _finish_hunting 0 2024-05-15T12:43:13.114 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.123+0000 7f8a79b45640 1 monclient: found mon.noname-b 2024-05-15T12:43:13.114 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.123+0000 7f8a79b45640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:13.114 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.123+0000 7f8a79b45640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.117:3301/0 2024-05-15T12:43:13.114 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.123+0000 7f8a79b45640 1 -- 172.21.15.117:0/1543042612 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a740ae3b0 con 0x7f8a740af1c0 2024-05-15T12:43:13.114 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.125+0000 7f8a79b45640 1 --2- 172.21.15.117:0/1543042612 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8a740af1c0 0x7f8a74142c70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f8a64009870 tx=0x7f8a6402b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bce84ff6fc15515c server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:13.118 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.129+0000 7f8a79344640 1 -- 172.21.15.117:0/1543042612 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a640317c0 con 0x7f8a740af1c0 2024-05-15T12:43:13.118 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.129+0000 7f8a79344640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:13.118 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.129+0000 7f8a79344640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:43:13.118 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.129+0000 7f8a79344640 10 monclient: dump: 2024-05-15T12:43:13.119 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:13.119 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:13.119 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.119 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.119 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:13.119 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:13.119 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:13.119 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:13.119 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:13.119 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:13.119 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.129+0000 7f8a79344640 10 monclient: _finish_auth 0 2024-05-15T12:43:13.119 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.129+0000 7f8a79344640 10 monclient: _check_auth_tickets 2024-05-15T12:43:13.119 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.129+0000 7f8a79344640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:13.119 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.129+0000 7f8a79344640 1 -- 172.21.15.117:0/1543042612 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a64037070 con 0x7f8a740af1c0 2024-05-15T12:43:13.119 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.129+0000 7f8a79344640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:13.120 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.129+0000 7f8a79344640 1 -- 172.21.15.117:0/1543042612 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a64031b80 con 0x7f8a740af1c0 2024-05-15T12:43:13.120 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.129+0000 7f8a7c5aa640 5 monclient: authenticate success, global_id 4307 2024-05-15T12:43:13.120 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.129+0000 7f8a7c5aa640 10 monclient: get_monmap_and_config success 2024-05-15T12:43:13.120 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.129+0000 7f8a7c5aa640 4 set_mon_vals no callback set 2024-05-15T12:43:13.120 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.129+0000 7f8a79344640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:13.120 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.129+0000 7f8a79344640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T12:43:13.120 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.129+0000 7f8a79344640 10 monclient: dump: 2024-05-15T12:43:13.120 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:13.120 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:13.120 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.120 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.120 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:13.120 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:13.120 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:13.120 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:13.121 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:13.121 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:13.124 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.130+0000 7f8a7c5aa640 10 monclient: shutdown 2024-05-15T12:43:13.124 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.130+0000 7f8a7c5aa640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:13.124 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.130+0000 7f8a7c5aa640 1 -- 172.21.15.117:0/1543042612 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8a740af1c0 msgr2=0x7f8a74142c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:13.124 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.130+0000 7f8a7c5aa640 1 --2- 172.21.15.117:0/1543042612 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8a740af1c0 0x7f8a74142c70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f8a64009870 tx=0x7f8a6402b860 comp rx=0 tx=0).stop 2024-05-15T12:43:13.124 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.130+0000 7f8a7c5aa640 1 -- 172.21.15.117:0/1543042612 shutdown_connections 2024-05-15T12:43:13.124 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.130+0000 7f8a7c5aa640 1 --2- 172.21.15.117:0/1543042612 >> v2:172.21.15.117:3300/0 conn(0x7f8a741431b0 0x7f8a74143660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:13.124 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.130+0000 7f8a7c5aa640 1 --2- 172.21.15.117:0/1543042612 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8a740af1c0 0x7f8a74142c70 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:13.125 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.130+0000 7f8a7c5aa640 1 -- 172.21.15.117:0/1543042612 >> 172.21.15.117:0/1543042612 conn(0x7f8a740b2ac0 msgr2=0x7f8a740b2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:13.125 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.130+0000 7f8a7c5aa640 1 -- 172.21.15.117:0/1543042612 shutdown_connections 2024-05-15T12:43:13.125 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.130+0000 7f8a7c5aa640 1 -- 172.21.15.117:0/1543042612 wait complete. 2024-05-15T12:43:13.125 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.131+0000 7f8a7c5aa640 10 monclient: build_initial_monmap 2024-05-15T12:43:13.125 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.131+0000 7f8a7c5aa640 10 monclient: monmap: 2024-05-15T12:43:13.125 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:13.125 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:43:13.125 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-05-15T12:43:13.125 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-05-15T12:43:13.125 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:13.125 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:13.125 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T12:43:13.125 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T12:43:13.125 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T12:43:13.126 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:13.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.131+0000 7f8a7c5aa640 1 librados: starting msgr at 2024-05-15T12:43:13.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.131+0000 7f8a7c5aa640 1 librados: starting objecter 2024-05-15T12:43:13.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.132+0000 7f8a7c5aa640 1 Processor -- start 2024-05-15T12:43:13.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.132+0000 7f8a7c5aa640 1 -- start start 2024-05-15T12:43:13.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.132+0000 7f8a7c5aa640 1 librados: setting wanted keys 2024-05-15T12:43:13.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.132+0000 7f8a7c5aa640 1 librados: calling monclient init 2024-05-15T12:43:13.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.132+0000 7f8a7c5aa640 10 monclient: init 2024-05-15T12:43:13.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.132+0000 7f8a7c5aa640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:13.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.132+0000 7f8a7c5aa640 10 monclient: _add_conns ranks=[2,1,0] 2024-05-15T12:43:13.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.132+0000 7f8a7c5aa640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8a7406c900 0x7f8a740b5be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:13.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.132+0000 7f8a7c5aa640 10 monclient(hunting): picked mon.noname-c con 0x7f8a7406c900 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:43:13.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.132+0000 7f8a7c5aa640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8a740af1c0 0x7f8a741d8050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:13.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.132+0000 7f8a7c5aa640 10 monclient(hunting): picked mon.noname-b con 0x7f8a740af1c0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:43:13.127 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.132+0000 7f8a7c5aa640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8a741431b0 0x7f8a7413de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:13.127 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.132+0000 7f8a7c5aa640 10 monclient(hunting): picked mon.noname-a con 0x7f8a741431b0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T12:43:13.127 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.132+0000 7f8a7c5aa640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:13.127 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.132+0000 7f8a7c5aa640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a740af5e0 con 0x7f8a741431b0 2024-05-15T12:43:13.127 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.132+0000 7f8a7c5aa640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:13.127 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.132+0000 7f8a7c5aa640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a7406d730 con 0x7f8a740af1c0 2024-05-15T12:43:13.127 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.132+0000 7f8a7c5aa640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:13.127 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.132+0000 7f8a7c5aa640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a740af760 con 0x7f8a7406c900 2024-05-15T12:43:13.127 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.132+0000 7f8a7c5aa640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:13.127 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.132+0000 7f8a7c5aa640 10 monclient(hunting): authenticate will time out at 1356.373441s 2024-05-15T12:43:13.127 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.133+0000 7f8a7ab47640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8a741431b0 0x7f8a7413de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:13.127 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.133+0000 7f8a7a346640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8a7406c900 0x7f8a740b5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:13.127 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.133+0000 7f8a7ab47640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8a741431b0 0x7f8a7413de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:60280/0 (socket says 172.21.15.117:60280) 2024-05-15T12:43:13.127 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.133+0000 7f8a7ab47640 1 -- 172.21.15.117:0/3102839906 learned_addr learned my addr 172.21.15.117:0/3102839906 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:43:13.127 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.133+0000 7f8a7ab47640 10 monclient(hunting): get_auth_request con 0x7f8a741431b0 auth_method 0 2024-05-15T12:43:13.128 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.133+0000 7f8a7ab47640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:13.128 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.133+0000 7f8a7ab47640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:13.128 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.133+0000 7f8a7ab47640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:13.128 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.133+0000 7f8a79b45640 1 --2- 172.21.15.117:0/3102839906 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8a740af1c0 0x7f8a741d8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:13.128 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.133+0000 7f8a7a346640 10 monclient(hunting): get_auth_request con 0x7f8a7406c900 auth_method 0 2024-05-15T12:43:13.128 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.133+0000 7f8a7a346640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:13.128 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.133+0000 7f8a7a346640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:13.128 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.133+0000 7f8a7a346640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:13.128 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.133+0000 7f8a79b45640 10 monclient(hunting): get_auth_request con 0x7f8a740af1c0 auth_method 0 2024-05-15T12:43:13.128 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.133+0000 7f8a79b45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:13.128 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.133+0000 7f8a79b45640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:13.128 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.133+0000 7f8a79b45640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:13.128 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.134+0000 7f8a7a346640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:13.128 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.134+0000 7f8a7a346640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:13.128 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.134+0000 7f8a7a346640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:13.129 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.134+0000 7f8a7ab47640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:13.129 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.134+0000 7f8a7ab47640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:13.129 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.134+0000 7f8a7ab47640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:13.129 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.134+0000 7f8a79b45640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:13.129 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.134+0000 7f8a79b45640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:13.129 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.134+0000 7f8a79b45640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:13.130 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.137+0000 7f8a7a346640 10 monclient(hunting): handle_auth_done global_id 4310 payload 824 2024-05-15T12:43:13.130 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.137+0000 7f8a7a346640 1 -- 172.21.15.117:0/3102839906 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8a740af1c0 msgr2=0x7f8a741d8050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:13.130 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.137+0000 7f8a7a346640 1 --2- 172.21.15.117:0/3102839906 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8a740af1c0 0x7f8a741d8050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:13.130 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.137+0000 7f8a7a346640 1 -- 172.21.15.117:0/3102839906 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8a741431b0 msgr2=0x7f8a7413de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:13.130 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.137+0000 7f8a7a346640 1 --2- 172.21.15.117:0/3102839906 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8a741431b0 0x7f8a7413de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:13.130 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.137+0000 7f8a7a346640 10 monclient: _finish_hunting 0 2024-05-15T12:43:13.130 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.137+0000 7f8a7a346640 1 monclient: found mon.noname-c 2024-05-15T12:43:13.130 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.137+0000 7f8a7a346640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:13.130 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.137+0000 7f8a7a346640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.117:3301/0 2024-05-15T12:43:13.131 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.137+0000 7f8a7a346640 1 -- 172.21.15.117:0/3102839906 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a741d89b0 con 0x7f8a7406c900 2024-05-15T12:43:13.131 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.137+0000 7f8a7ab47640 1 --2- 172.21.15.117:0/3102839906 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8a741431b0 0x7f8a7413de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-15T12:43:13.131 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.139+0000 7f8a7a346640 1 --2- 172.21.15.117:0/3102839906 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8a7406c900 0x7f8a740b5be0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8a7000c370 tx=0x7f8a7000c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:13.131 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.139+0000 7f8a6affd640 10 client.?.objecter ms_handle_connect 0x7f8a7406c900 2024-05-15T12:43:13.131 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.139+0000 7f8a6affd640 10 client.?.objecter resend_mon_ops 2024-05-15T12:43:13.132 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.142+0000 7f8a6affd640 1 -- 172.21.15.117:0/3102839906 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a700052d0 con 0x7f8a7406c900 2024-05-15T12:43:13.132 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.142+0000 7f8a6affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:13.132 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.142+0000 7f8a6affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:43:13.132 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.142+0000 7f8a6affd640 10 monclient: dump: 2024-05-15T12:43:13.132 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:13.132 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:13.132 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.132 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.132 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:13.132 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:13.132 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:13.133 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:13.133 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:13.133 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:13.133 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.142+0000 7f8a6affd640 10 monclient: _finish_auth 0 2024-05-15T12:43:13.133 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.142+0000 7f8a6affd640 10 monclient: _check_auth_tickets 2024-05-15T12:43:13.133 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.142+0000 7f8a6affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:13.133 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.142+0000 7f8a6affd640 1 -- 172.21.15.117:0/3102839906 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a70005490 con 0x7f8a7406c900 2024-05-15T12:43:13.133 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.142+0000 7f8a6affd640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:13.133 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.142+0000 7f8a6affd640 1 -- 172.21.15.117:0/3102839906 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a70007260 con 0x7f8a7406c900 2024-05-15T12:43:13.133 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.142+0000 7f8a6affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:13.133 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.142+0000 7f8a6affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T12:43:13.133 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.142+0000 7f8a6affd640 10 monclient: dump: 2024-05-15T12:43:13.133 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:13.133 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:13.133 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.134 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.134 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:13.134 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:13.134 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:13.134 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:13.134 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:13.134 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:13.134 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.142+0000 7f8a7c5aa640 5 monclient: authenticate success, global_id 4310 2024-05-15T12:43:13.134 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.142+0000 7f8a7c5aa640 10 monclient: _renew_subs 2024-05-15T12:43:13.134 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.143+0000 7f8a7c5aa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T12:43:13.134 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.143+0000 7f8a7c5aa640 1 -- 172.21.15.117:0/3102839906 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a7411ef20 con 0x7f8a7406c900 2024-05-15T12:43:13.134 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.142+0000 7f8a78b43640 4 set_mon_vals no callback set 2024-05-15T12:43:13.134 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.143+0000 7f8a7c5aa640 10 client.4310.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:43:13.134 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.143+0000 7f8a7c5aa640 10 monclient: _renew_subs 2024-05-15T12:43:13.135 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.143+0000 7f8a7c5aa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T12:43:13.135 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.143+0000 7f8a7c5aa640 1 -- 172.21.15.117:0/3102839906 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a7413e3d0 con 0x7f8a7406c900 2024-05-15T12:43:13.135 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.143+0000 7f8a7c5aa640 1 librados: init done 2024-05-15T12:43:13.135 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.143+0000 7f8a7c5aa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:13.135 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.143+0000 7f8a7c5aa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:13.135 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.143+0000 7f8a7c5aa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T12:43:13.135 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.143+0000 7f8a7c5aa640 1 -- 172.21.15.117:0/3102839906 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a741d87d0 con 0x7f8a7406c900 2024-05-15T12:43:13.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.164+0000 7f8a6affd640 1 -- 172.21.15.117:0/3102839906 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f8a70007400 con 0x7f8a7406c900 2024-05-15T12:43:13.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.164+0000 7f8a6affd640 1 --2- 172.21.15.117:0/3102839906 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f8a5806ce60 0x7f8a5806f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:13.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.164+0000 7f8a79b45640 1 --2- 172.21.15.117:0/3102839906 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f8a5806ce60 0x7f8a5806f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:13.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.164+0000 7f8a79b45640 10 monclient: get_auth_request con 0x7f8a5806ce60 auth_method 0 2024-05-15T12:43:13.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.165+0000 7f8a6affd640 10 client.4310.objecter ms_handle_connect 0x7f8a5806ce60 2024-05-15T12:43:13.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.165+0000 7f8a79b45640 1 --2- 172.21.15.117:0/3102839906 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f8a5806ce60 0x7f8a5806f320 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8a64036000 tx=0x7f8a6403d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:13.156 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.167+0000 7f8a6affd640 1 -- 172.21.15.117:0/3102839906 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f8a70056380 con 0x7f8a7406c900 2024-05-15T12:43:13.157 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.167+0000 7f8a6affd640 10 client.4310.objecter ms_dispatch 0x7f8a74000b90 osd_map(26..26 src has 1..26) v4 2024-05-15T12:43:13.157 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.168+0000 7f8a6affd640 3 client.4310.objecter handle_osd_map got epochs [26,26] > 0 2024-05-15T12:43:13.157 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.168+0000 7f8a6affd640 3 client.4310.objecter handle_osd_map decoding full epoch 26 2024-05-15T12:43:13.157 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.168+0000 7f8a6affd640 20 client.4310.objecter dump_active .. 0 homeless 2024-05-15T12:43:13.185 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-05-15T12:43:13.210 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-05-15T12:43:13.287 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.296+0000 7f2b60b87640 10 monclient: get_monmap_and_config 2024-05-15T12:43:13.287 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.296+0000 7f2b60b87640 10 monclient: build_initial_monmap 2024-05-15T12:43:13.287 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.296+0000 7f2b60b87640 10 monclient: monmap: 2024-05-15T12:43:13.287 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:13.287 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:13.287 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:43:13.297619+0000 2024-05-15T12:43:13.287 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:43:13.297619+0000 2024-05-15T12:43:13.288 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:13.288 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:13.288 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T12:43:13.288 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T12:43:13.288 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T12:43:13.288 INFO:teuthology.orchestra.run.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T12:43:13.288 INFO:teuthology.orchestra.run.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T12:43:13.288 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:13.288 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.297+0000 7f2b60b87640 1 Processor -- start 2024-05-15T12:43:13.289 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.297+0000 7f2b60b87640 1 -- start start 2024-05-15T12:43:13.289 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.297+0000 7f2b60b87640 10 monclient: init 2024-05-15T12:43:13.291 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.297+0000 7f2b60b87640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:13.291 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.297+0000 7f2b60b87640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-05-15T12:43:13.291 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.297+0000 7f2b60b87640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f2b5c0b86e0 0x7f2b5c0aa0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:13.291 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.297+0000 7f2b60b87640 10 monclient(hunting): picked mon.noname-c con 0x7f2b5c0b86e0 addr v2:172.21.15.178:3300/0 2024-05-15T12:43:13.291 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.297+0000 7f2b60b87640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f2b5c0b1dd0 0x7f2b5c145bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:13.291 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.297+0000 7f2b60b87640 10 monclient(hunting): picked mon.noname-a con 0x7f2b5c0b1dd0 addr v2:172.21.15.117:3300/0 2024-05-15T12:43:13.291 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.297+0000 7f2b60b87640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2b5c13f870 0x7f2b5c13fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:13.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.297+0000 7f2b60b87640 10 monclient(hunting): picked mon.noname-b con 0x7f2b5c13f870 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:43:13.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.297+0000 7f2b60b87640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:13.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.298+0000 7f2b60b87640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b5c0b8ac0 con 0x7f2b5c0b1dd0 2024-05-15T12:43:13.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.298+0000 7f2b60b87640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:13.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.298+0000 7f2b60b87640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b5c0aa620 con 0x7f2b5c0b86e0 2024-05-15T12:43:13.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.298+0000 7f2b60b87640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:13.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.298+0000 7f2b60b87640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5c0a9bc0 con 0x7f2b5c13f870 2024-05-15T12:43:13.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.298+0000 7f2b60b87640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:13.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.298+0000 7f2b60b87640 10 monclient(hunting): authenticate will time out at 1356.539820s 2024-05-15T12:43:13.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.298+0000 7f2b59d9b640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f2b5c0b1dd0 0x7f2b5c145bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:13.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.298+0000 7f2b5a59c640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f2b5c0b86e0 0x7f2b5c0aa0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:13.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.298+0000 7f2b5a59c640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f2b5c0b86e0 0x7f2b5c0aa0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:42588/0 (socket says 172.21.15.117:42588) 2024-05-15T12:43:13.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.298+0000 7f2b5a59c640 1 -- 172.21.15.117:0/3965970757 learned_addr learned my addr 172.21.15.117:0/3965970757 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:43:13.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.298+0000 7f2b5a59c640 10 monclient(hunting): get_auth_request con 0x7f2b5c0b86e0 auth_method 0 2024-05-15T12:43:13.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.298+0000 7f2b5a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:13.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.298+0000 7f2b5a59c640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:13.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.298+0000 7f2b5a59c640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:13.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.299+0000 7f2b59d9b640 10 monclient(hunting): get_auth_request con 0x7f2b5c0b1dd0 auth_method 0 2024-05-15T12:43:13.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.299+0000 7f2b59d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:13.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.299+0000 7f2b59d9b640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:13.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.299+0000 7f2b59d9b640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:13.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.299+0000 7f2b5a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:13.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.299+0000 7f2b5a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:13.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.299+0000 7f2b5a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:13.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.299+0000 7f2b59d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:13.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.299+0000 7f2b59d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:13.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.299+0000 7f2b59d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:13.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.300+0000 7f2b5a59c640 10 monclient(hunting): handle_auth_done global_id 4363 payload 274 2024-05-15T12:43:13.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.300+0000 7f2b5a59c640 1 -- 172.21.15.117:0/3965970757 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2b5c13f870 msgr2=0x7f2b5c13fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:43:13.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.300+0000 7f2b5a59c640 1 --2- 172.21.15.117:0/3965970757 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2b5c13f870 0x7f2b5c13fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:13.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.300+0000 7f2b5a59c640 1 -- 172.21.15.117:0/3965970757 >> v2:172.21.15.117:3300/0 conn(0x7f2b5c0b1dd0 msgr2=0x7f2b5c145bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:43:13.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.300+0000 7f2b5a59c640 1 --2- 172.21.15.117:0/3965970757 >> v2:172.21.15.117:3300/0 conn(0x7f2b5c0b1dd0 0x7f2b5c145bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:13.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.301+0000 7f2b5a59c640 10 monclient: _finish_hunting 0 2024-05-15T12:43:13.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.301+0000 7f2b5a59c640 1 monclient: found mon.noname-c 2024-05-15T12:43:13.295 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.301+0000 7f2b5a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:13.295 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.301+0000 7f2b5a59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.178:3300/0 2024-05-15T12:43:13.295 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.301+0000 7f2b5a59c640 1 -- 172.21.15.117:0/3965970757 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b5c146100 con 0x7f2b5c0b86e0 2024-05-15T12:43:13.296 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.303+0000 7f2b5a59c640 1 --2- 172.21.15.117:0/3965970757 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2b5c0b86e0 0x7f2b5c0aa0e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c0b8d50 tx=0x7f2b4c02b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1694d917b21e61c1 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:13.296 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.305+0000 7f2b5959a640 1 -- 172.21.15.117:0/3965970757 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b4c02e070 con 0x7f2b5c0b86e0 2024-05-15T12:43:13.296 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.305+0000 7f2b5959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:13.296 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.305+0000 7f2b5959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:43:13.296 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.305+0000 7f2b5959a640 10 monclient: dump: 2024-05-15T12:43:13.296 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:13.296 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:13.296 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.296 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.296 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:13.297 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:13.297 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:13.297 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:13.297 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:13.297 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:13.297 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.305+0000 7f2b5959a640 10 monclient: _finish_auth 0 2024-05-15T12:43:13.297 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.305+0000 7f2b5959a640 10 monclient: _check_auth_tickets 2024-05-15T12:43:13.297 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.305+0000 7f2b5959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:13.297 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.305+0000 7f2b60b87640 5 monclient: authenticate success, global_id 4363 2024-05-15T12:43:13.297 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.305+0000 7f2b60b87640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:43:13.301 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.306+0000 7f2b5959a640 1 -- 172.21.15.117:0/3965970757 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b4c037070 con 0x7f2b5c0b86e0 2024-05-15T12:43:13.301 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.306+0000 7f2b5959a640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:13.301 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.306+0000 7f2b5959a640 1 -- 172.21.15.117:0/3965970757 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b4c031a00 con 0x7f2b5c0b86e0 2024-05-15T12:43:13.301 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.306+0000 7f2b5959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:13.301 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.306+0000 7f2b5959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:43:13.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.306+0000 7f2b5959a640 10 monclient: dump: 2024-05-15T12:43:13.302 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:13.302 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:13.302 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.302 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.302 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:13.302 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:13.302 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:13.302 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:13.302 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:13.302 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:13.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.307+0000 7f2b60b87640 10 monclient: get_monmap_and_config success 2024-05-15T12:43:13.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.307+0000 7f2b60b87640 4 set_mon_vals no callback set 2024-05-15T12:43:13.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.307+0000 7f2b60b87640 10 monclient: shutdown 2024-05-15T12:43:13.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.307+0000 7f2b60b87640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:13.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.307+0000 7f2b60b87640 1 -- 172.21.15.117:0/3965970757 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2b5c0b86e0 msgr2=0x7f2b5c0aa0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:13.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.307+0000 7f2b60b87640 1 --2- 172.21.15.117:0/3965970757 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2b5c0b86e0 0x7f2b5c0aa0e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c0b8d50 tx=0x7f2b4c02b880 comp rx=0 tx=0).stop 2024-05-15T12:43:13.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.307+0000 7f2b60b87640 1 -- 172.21.15.117:0/3965970757 shutdown_connections 2024-05-15T12:43:13.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.307+0000 7f2b60b87640 1 --2- 172.21.15.117:0/3965970757 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2b5c13f870 0x7f2b5c13fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:13.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.307+0000 7f2b60b87640 1 --2- 172.21.15.117:0/3965970757 >> v2:172.21.15.117:3300/0 conn(0x7f2b5c0b1dd0 0x7f2b5c145bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:13.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.307+0000 7f2b60b87640 1 --2- 172.21.15.117:0/3965970757 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2b5c0b86e0 0x7f2b5c0aa0e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:13.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.307+0000 7f2b60b87640 1 -- 172.21.15.117:0/3965970757 >> 172.21.15.117:0/3965970757 conn(0x7f2b5c066f00 msgr2=0x7f2b5c067310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:13.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.307+0000 7f2b60b87640 1 -- 172.21.15.117:0/3965970757 shutdown_connections 2024-05-15T12:43:13.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.307+0000 7f2b60b87640 1 -- 172.21.15.117:0/3965970757 wait complete. 2024-05-15T12:43:13.304 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.308+0000 7f2b60b87640 10 monclient: build_initial_monmap 2024-05-15T12:43:13.304 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.308+0000 7f2b60b87640 10 monclient: monmap: 2024-05-15T12:43:13.304 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:13.304 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:43:13.304 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-05-15T12:43:13.304 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-05-15T12:43:13.304 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:13.304 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:13.304 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T12:43:13.304 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T12:43:13.304 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T12:43:13.304 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:13.305 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.308+0000 7f2b60b87640 1 librados: starting msgr at 2024-05-15T12:43:13.305 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.308+0000 7f2b60b87640 1 librados: starting objecter 2024-05-15T12:43:13.305 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.309+0000 7f2b60b87640 1 Processor -- start 2024-05-15T12:43:13.305 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.309+0000 7f2b60b87640 1 -- start start 2024-05-15T12:43:13.305 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.309+0000 7f2b60b87640 1 librados: setting wanted keys 2024-05-15T12:43:13.305 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.309+0000 7f2b60b87640 1 librados: calling monclient init 2024-05-15T12:43:13.305 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.309+0000 7f2b60b87640 10 monclient: init 2024-05-15T12:43:13.305 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.309+0000 7f2b60b87640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:13.305 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.309+0000 7f2b60b87640 10 monclient: _add_conns ranks=[2,1,0] 2024-05-15T12:43:13.305 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.309+0000 7f2b60b87640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2b5c0b1dd0 0x7f2b5c145360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:13.306 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.309+0000 7f2b60b87640 10 monclient(hunting): picked mon.noname-c con 0x7f2b5c0b1dd0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:43:13.306 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.309+0000 7f2b60b87640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2b5c0b86e0 0x7f2b5c1429f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:13.306 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.309+0000 7f2b60b87640 10 monclient(hunting): picked mon.noname-b con 0x7f2b5c0b86e0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:43:13.306 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.309+0000 7f2b60b87640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b5c13f870 0x7f2b5c142f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:13.306 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.309+0000 7f2b60b87640 10 monclient(hunting): picked mon.noname-a con 0x7f2b5c13f870 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T12:43:13.306 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.309+0000 7f2b60b87640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:13.306 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.309+0000 7f2b60b87640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5c0b3a60 con 0x7f2b5c13f870 2024-05-15T12:43:13.306 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.309+0000 7f2b60b87640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:13.306 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.309+0000 7f2b60b87640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5c1401d0 con 0x7f2b5c0b86e0 2024-05-15T12:43:13.306 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.309+0000 7f2b60b87640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:13.306 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.309+0000 7f2b60b87640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5c0b3be0 con 0x7f2b5c0b1dd0 2024-05-15T12:43:13.306 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.309+0000 7f2b60b87640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:13.307 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.309+0000 7f2b60b87640 10 monclient(hunting): authenticate will time out at 1356.551436s 2024-05-15T12:43:13.307 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.310+0000 7f2b5ad9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b5c13f870 0x7f2b5c142f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:13.307 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.310+0000 7f2b59d9b640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2b5c0b86e0 0x7f2b5c1429f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:13.307 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.310+0000 7f2b59d9b640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2b5c0b86e0 0x7f2b5c1429f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:42600/0 (socket says 172.21.15.117:42600) 2024-05-15T12:43:13.307 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.310+0000 7f2b59d9b640 1 -- 172.21.15.117:0/794198670 learned_addr learned my addr 172.21.15.117:0/794198670 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:43:13.307 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.310+0000 7f2b59d9b640 10 monclient(hunting): get_auth_request con 0x7f2b5c0b86e0 auth_method 0 2024-05-15T12:43:13.307 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.310+0000 7f2b59d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:13.307 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.310+0000 7f2b59d9b640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:13.307 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.310+0000 7f2b59d9b640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:13.307 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.310+0000 7f2b5ad9d640 10 monclient(hunting): get_auth_request con 0x7f2b5c13f870 auth_method 0 2024-05-15T12:43:13.308 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.310+0000 7f2b5ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:13.308 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.310+0000 7f2b5ad9d640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:13.308 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.310+0000 7f2b5ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:13.308 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.310+0000 7f2b59d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:13.308 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.310+0000 7f2b59d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:13.308 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.311+0000 7f2b59d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:13.308 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.311+0000 7f2b5ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:13.308 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.311+0000 7f2b5ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:13.308 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.311+0000 7f2b5ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:13.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.313+0000 7f2b59d9b640 10 monclient(hunting): handle_auth_done global_id 4366 payload 824 2024-05-15T12:43:13.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.313+0000 7f2b59d9b640 1 -- 172.21.15.117:0/794198670 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2b5c0b1dd0 msgr2=0x7f2b5c145360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:13.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.313+0000 7f2b59d9b640 1 --2- 172.21.15.117:0/794198670 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2b5c0b1dd0 0x7f2b5c145360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:13.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.313+0000 7f2b59d9b640 1 -- 172.21.15.117:0/794198670 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b5c13f870 msgr2=0x7f2b5c142f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:13.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.313+0000 7f2b59d9b640 1 --2- 172.21.15.117:0/794198670 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b5c13f870 0x7f2b5c142f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:13.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.313+0000 7f2b59d9b640 10 monclient: _finish_hunting 0 2024-05-15T12:43:13.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.313+0000 7f2b59d9b640 1 monclient: found mon.noname-b 2024-05-15T12:43:13.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.313+0000 7f2b59d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:13.312 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.313+0000 7f2b59d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2024-05-15T12:43:13.312 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.313+0000 7f2b59d9b640 1 -- 172.21.15.117:0/794198670 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b5c1d7b70 con 0x7f2b5c0b86e0 2024-05-15T12:43:13.312 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.316+0000 7f2b59d9b640 1 --2- 172.21.15.117:0/794198670 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2b5c0b86e0 0x7f2b5c1429f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2b5000e9b0 tx=0x7f2b5000ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:13.312 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.316+0000 7f2b4affd640 10 client.?.objecter ms_handle_connect 0x7f2b5c0b86e0 2024-05-15T12:43:13.312 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.316+0000 7f2b4affd640 10 client.?.objecter resend_mon_ops 2024-05-15T12:43:13.312 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.319+0000 7f2b4affd640 1 -- 172.21.15.117:0/794198670 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b50017070 con 0x7f2b5c0b86e0 2024-05-15T12:43:13.312 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.319+0000 7f2b4affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:13.312 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.319+0000 7f2b4affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:43:13.312 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.319+0000 7f2b4affd640 10 monclient: dump: 2024-05-15T12:43:13.312 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:13.313 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:13.313 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.313 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.313 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:13.313 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:13.313 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:13.313 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:13.313 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:13.313 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:13.313 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.319+0000 7f2b4affd640 10 monclient: _finish_auth 0 2024-05-15T12:43:13.313 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.319+0000 7f2b4affd640 10 monclient: _check_auth_tickets 2024-05-15T12:43:13.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.319+0000 7f2b4affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:13.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.319+0000 7f2b4affd640 1 -- 172.21.15.117:0/794198670 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b5000c9a0 con 0x7f2b5c0b86e0 2024-05-15T12:43:13.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.319+0000 7f2b60b87640 5 monclient: authenticate success, global_id 4366 2024-05-15T12:43:13.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.319+0000 7f2b60b87640 10 monclient: _renew_subs 2024-05-15T12:43:13.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.319+0000 7f2b60b87640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:43:13.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.319+0000 7f2b60b87640 1 -- 172.21.15.117:0/794198670 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b5c1d7e00 con 0x7f2b5c0b86e0 2024-05-15T12:43:13.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.319+0000 7f2b4affd640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:13.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.319+0000 7f2b4affd640 1 -- 172.21.15.117:0/794198670 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b5000cd90 con 0x7f2b5c0b86e0 2024-05-15T12:43:13.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.319+0000 7f2b4affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:13.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.319+0000 7f2b4affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:43:13.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.319+0000 7f2b4affd640 10 monclient: dump: 2024-05-15T12:43:13.314 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:13.315 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:13.315 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.315 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.315 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:13.315 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:13.315 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:13.315 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:13.315 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:13.315 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:13.315 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.319+0000 7f2b60b87640 10 client.4366.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:43:13.315 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.319+0000 7f2b60b87640 10 monclient: _renew_subs 2024-05-15T12:43:13.315 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.319+0000 7f2b58d99640 4 set_mon_vals no callback set 2024-05-15T12:43:13.316 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.319+0000 7f2b60b87640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:43:13.316 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.319+0000 7f2b60b87640 1 -- 172.21.15.117:0/794198670 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b5c1d8350 con 0x7f2b5c0b86e0 2024-05-15T12:43:13.316 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.319+0000 7f2b60b87640 1 librados: init done 2024-05-15T12:43:13.316 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.320+0000 7f2b60b87640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:13.316 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.320+0000 7f2b60b87640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:13.316 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.320+0000 7f2b60b87640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:43:13.316 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.320+0000 7f2b60b87640 1 -- 172.21.15.117:0/794198670 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b5c04f0e0 con 0x7f2b5c0b86e0 2024-05-15T12:43:13.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.338+0000 7f2b4affd640 1 -- 172.21.15.117:0/794198670 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f2b5001a070 con 0x7f2b5c0b86e0 2024-05-15T12:43:13.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.339+0000 7f2b4affd640 1 --2- 172.21.15.117:0/794198670 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f2b3c06ceb0 0x7f2b3c06f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:13.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.339+0000 7f2b5a59c640 1 --2- 172.21.15.117:0/794198670 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f2b3c06ceb0 0x7f2b3c06f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:13.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.339+0000 7f2b5a59c640 10 monclient: get_auth_request con 0x7f2b3c06ceb0 auth_method 0 2024-05-15T12:43:13.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.339+0000 7f2b4affd640 10 client.4366.objecter ms_handle_connect 0x7f2b3c06ceb0 2024-05-15T12:43:13.330 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.339+0000 7f2b5a59c640 1 --2- 172.21.15.117:0/794198670 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f2b3c06ceb0 0x7f2b3c06f370 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c036000 tx=0x7f2b4c030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:13.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.341+0000 7f139d304640 10 monclient: get_monmap_and_config 2024-05-15T12:43:13.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.341+0000 7f139d304640 10 monclient: build_initial_monmap 2024-05-15T12:43:13.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.341+0000 7f139d304640 10 monclient: monmap: 2024-05-15T12:43:13.332 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:13.332 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:13.332 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:43:13.342438+0000 2024-05-15T12:43:13.332 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:43:13.342438+0000 2024-05-15T12:43:13.332 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:13.332 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:13.332 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T12:43:13.332 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T12:43:13.332 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T12:43:13.332 INFO:teuthology.orchestra.run.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T12:43:13.332 INFO:teuthology.orchestra.run.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T12:43:13.332 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:13.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.341+0000 7f139d304640 1 Processor -- start 2024-05-15T12:43:13.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.342+0000 7f139d304640 1 -- start start 2024-05-15T12:43:13.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.342+0000 7f139d304640 10 monclient: init 2024-05-15T12:43:13.333 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-05-15T12:43:13.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.342+0000 7f2b4affd640 1 -- 172.21.15.117:0/794198670 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f2b50057220 con 0x7f2b5c0b86e0 2024-05-15T12:43:13.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.342+0000 7f2b4affd640 10 client.4366.objecter ms_dispatch 0x7f2b5c000b90 osd_map(26..26 src has 1..26) v4 2024-05-15T12:43:13.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.342+0000 7f2b4affd640 3 client.4366.objecter handle_osd_map got epochs [26,26] > 0 2024-05-15T12:43:13.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.342+0000 7f2b4affd640 3 client.4366.objecter handle_osd_map decoding full epoch 26 2024-05-15T12:43:13.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.342+0000 7f2b4affd640 20 client.4366.objecter dump_active .. 0 homeless 2024-05-15T12:43:13.336 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.342+0000 7f139d304640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:13.336 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.342+0000 7f139d304640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-05-15T12:43:13.336 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.342+0000 7f139d304640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f13980b4d00 0x7f13980b1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:13.336 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.342+0000 7f139d304640 10 monclient(hunting): picked mon.noname-a con 0x7f13980b4d00 addr v2:172.21.15.117:3300/0 2024-05-15T12:43:13.336 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.342+0000 7f139d304640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f13980b50e0 addr v1:172.21.15.117:6789/0 2024-05-15T12:43:13.336 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.342+0000 7f139d304640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f13980a94b0 0x7f13980b0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:13.336 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.342+0000 7f139d304640 10 monclient(hunting): picked mon.noname-b con 0x7f13980a94b0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:43:13.336 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.342+0000 7f139d304640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13980ab1b0 con 0x7f13980b50e0 2024-05-15T12:43:13.336 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.342+0000 7f139d304640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:13.337 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.342+0000 7f139d304640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13980b23c0 con 0x7f13980b4d00 2024-05-15T12:43:13.337 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.342+0000 7f139d304640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:13.337 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.342+0000 7f139d304640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13980b1520 con 0x7f13980a94b0 2024-05-15T12:43:13.337 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.342+0000 7f139d304640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:13.337 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.342+0000 7f139d304640 10 monclient(hunting): authenticate will time out at 1356.584577s 2024-05-15T12:43:13.337 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.343+0000 7f1396ffd640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f13980b4d00 0x7f13980b1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:13.337 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.343+0000 7f13967fc640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f13980b50e0 0x7f1398145c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:48724/0 (socket says 172.21.15.117:48724) 2024-05-15T12:43:13.337 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.343+0000 7f13967fc640 1 -- 172.21.15.117:0/2211128526 learned_addr learned my addr 172.21.15.117:0/2211128526 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-05-15T12:43:13.337 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.344+0000 7f1396ffd640 10 monclient(hunting): get_auth_request con 0x7f13980b4d00 auth_method 0 2024-05-15T12:43:13.337 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.344+0000 7f1396ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:13.337 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.344+0000 7f1396ffd640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:13.337 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.344+0000 7f1396ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:13.337 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.345+0000 7f1396ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:13.337 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.345+0000 7f1396ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:13.337 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.345+0000 7f1396ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:13.339 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.347+0000 7f1396ffd640 10 monclient(hunting): handle_auth_done global_id 4356 payload 274 2024-05-15T12:43:13.339 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.347+0000 7f1396ffd640 1 -- 172.21.15.117:0/2211128526 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f13980a94b0 msgr2=0x7f13980b0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:43:13.339 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.347+0000 7f1396ffd640 1 --2- 172.21.15.117:0/2211128526 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f13980a94b0 0x7f13980b0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:13.339 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.347+0000 7f1396ffd640 1 -- 172.21.15.117:0/2211128526 >> v1:172.21.15.117:6789/0 conn(0x7f13980b50e0 legacy=0x7f1398145c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:13.339 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.347+0000 7f1396ffd640 10 monclient: _finish_hunting 0 2024-05-15T12:43:13.339 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.347+0000 7f1396ffd640 1 monclient: found mon.noname-a 2024-05-15T12:43:13.339 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.347+0000 7f1396ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:13.339 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.347+0000 7f1396ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.117:3300/0 2024-05-15T12:43:13.339 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.347+0000 7f1396ffd640 1 -- 172.21.15.117:0/2211128526 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13980b16d0 con 0x7f13980b4d00 2024-05-15T12:43:13.341 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.351+0000 7f1396ffd640 1 --2- 172.21.15.117:0/2211128526 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13980b4d00 0x7f13980b1e80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f13880097a0 tx=0x7f138802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=104441bba15e4509 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:13.346 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.356+0000 7f1395ffb640 1 -- 172.21.15.117:0/2211128526 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13880315a0 con 0x7f13980b4d00 2024-05-15T12:43:13.347 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.356+0000 7f1395ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:13.347 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.356+0000 7f1395ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T12:43:13.347 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.356+0000 7f1395ffb640 10 monclient: dump: 2024-05-15T12:43:13.347 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:13.347 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:13.347 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.347 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.347 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:13.347 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:13.347 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:13.347 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:13.347 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:13.347 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:13.347 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.356+0000 7f1395ffb640 10 monclient: _finish_auth 0 2024-05-15T12:43:13.347 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.356+0000 7f1395ffb640 10 monclient: _check_auth_tickets 2024-05-15T12:43:13.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.356+0000 7f1395ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:13.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.356+0000 7f1395ffb640 1 -- 172.21.15.117:0/2211128526 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1388037070 con 0x7f13980b4d00 2024-05-15T12:43:13.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.356+0000 7f1395ffb640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:13.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.356+0000 7f1395ffb640 1 -- 172.21.15.117:0/2211128526 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1388031960 con 0x7f13980b4d00 2024-05-15T12:43:13.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.356+0000 7f139d304640 5 monclient: authenticate success, global_id 4356 2024-05-15T12:43:13.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.356+0000 7f139d304640 10 monclient: get_monmap_and_config success 2024-05-15T12:43:13.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.356+0000 7f139d304640 4 set_mon_vals no callback set 2024-05-15T12:43:13.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.356+0000 7f1395ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:13.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.356+0000 7f1395ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T12:43:13.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.356+0000 7f1395ffb640 10 monclient: dump: 2024-05-15T12:43:13.348 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:13.348 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:13.348 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.348 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.349 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:13.349 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:13.349 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:13.349 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:13.349 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:13.349 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:13.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.357+0000 7f139d304640 10 monclient: shutdown 2024-05-15T12:43:13.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.357+0000 7f139d304640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:13.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.357+0000 7f139d304640 1 -- 172.21.15.117:0/2211128526 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13980b4d00 msgr2=0x7f13980b1e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:13.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.357+0000 7f139d304640 1 --2- 172.21.15.117:0/2211128526 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13980b4d00 0x7f13980b1e80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f13880097a0 tx=0x7f138802b840 comp rx=0 tx=0).stop 2024-05-15T12:43:13.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.357+0000 7f139d304640 1 -- 172.21.15.117:0/2211128526 shutdown_connections 2024-05-15T12:43:13.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.357+0000 7f139d304640 1 --2- 172.21.15.117:0/2211128526 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f13980a94b0 0x7f13980b0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:13.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.357+0000 7f139d304640 1 --2- 172.21.15.117:0/2211128526 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13980b4d00 0x7f13980b1e80 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:13.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.357+0000 7f139d304640 1 -- 172.21.15.117:0/2211128526 >> 172.21.15.117:0/2211128526 conn(0x7f1398066cc0 msgr2=0x7f13980670d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:13.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.357+0000 7f139d304640 1 -- 172.21.15.117:0/2211128526 shutdown_connections 2024-05-15T12:43:13.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.357+0000 7f139d304640 1 -- 172.21.15.117:0/2211128526 wait complete. 2024-05-15T12:43:13.353 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.358+0000 7f139d304640 10 monclient: build_initial_monmap 2024-05-15T12:43:13.353 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.358+0000 7f139d304640 10 monclient: monmap: 2024-05-15T12:43:13.353 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:13.353 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:43:13.353 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-05-15T12:43:13.353 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-05-15T12:43:13.353 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:13.353 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:13.353 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T12:43:13.353 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T12:43:13.353 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T12:43:13.353 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:13.353 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.358+0000 7f139d304640 1 librados: starting msgr at 2024-05-15T12:43:13.353 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.358+0000 7f139d304640 1 librados: starting objecter 2024-05-15T12:43:13.353 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.358+0000 7f139d304640 1 Processor -- start 2024-05-15T12:43:13.353 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.358+0000 7f139d304640 1 -- start start 2024-05-15T12:43:13.354 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.358+0000 7f139d304640 1 librados: setting wanted keys 2024-05-15T12:43:13.354 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.358+0000 7f139d304640 1 librados: calling monclient init 2024-05-15T12:43:13.354 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.358+0000 7f139d304640 10 monclient: init 2024-05-15T12:43:13.354 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.359+0000 7f139d304640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:13.354 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.359+0000 7f139d304640 10 monclient: _add_conns ranks=[2,1,0] 2024-05-15T12:43:13.354 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.359+0000 7f139d304640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f13980a94b0 0x7f13980abdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:13.354 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.359+0000 7f139d304640 10 monclient(hunting): picked mon.noname-c con 0x7f13980a94b0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:43:13.354 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.359+0000 7f139d304640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f13980b4d00 0x7f1398139190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:13.354 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.359+0000 7f139d304640 10 monclient(hunting): picked mon.noname-b con 0x7f13980b4d00 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:43:13.354 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.359+0000 7f139d304640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13980ae830 0x7f13981396d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:13.354 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.359+0000 7f139d304640 10 monclient(hunting): picked mon.noname-a con 0x7f13980ae830 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T12:43:13.354 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.359+0000 7f139d304640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:13.354 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.359+0000 7f139d304640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f139814d130 con 0x7f13980ae830 2024-05-15T12:43:13.354 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.359+0000 7f139d304640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:13.354 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.359+0000 7f139d304640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f139814cfb0 con 0x7f13980b4d00 2024-05-15T12:43:13.355 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.359+0000 7f139d304640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:13.355 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.359+0000 7f139d304640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f139814d2b0 con 0x7f13980a94b0 2024-05-15T12:43:13.355 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.359+0000 7f139d304640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:13.355 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.359+0000 7f139d304640 10 monclient(hunting): authenticate will time out at 1356.601348s 2024-05-15T12:43:13.355 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.360+0000 7f13977fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13980ae830 0x7f13981396d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:13.355 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.360+0000 7f13977fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13980ae830 0x7f13981396d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:60332/0 (socket says 172.21.15.117:60332) 2024-05-15T12:43:13.355 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.360+0000 7f13977fe640 1 -- 172.21.15.117:0/4276070227 learned_addr learned my addr 172.21.15.117:0/4276070227 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:43:13.355 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.360+0000 7f13977fe640 10 monclient(hunting): get_auth_request con 0x7f13980ae830 auth_method 0 2024-05-15T12:43:13.355 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.360+0000 7f13977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:13.355 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.360+0000 7f13977fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:13.355 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.360+0000 7f13977fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:13.355 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.360+0000 7f13977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:13.355 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.360+0000 7f13977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:13.356 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.361+0000 7f13977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:13.356 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.363+0000 7f13977fe640 10 monclient(hunting): handle_auth_done global_id 4362 payload 824 2024-05-15T12:43:13.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.364+0000 7f13977fe640 1 -- 172.21.15.117:0/4276070227 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f13980a94b0 msgr2=0x7f13980abdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:13.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.364+0000 7f13977fe640 1 --2- 172.21.15.117:0/4276070227 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f13980a94b0 0x7f13980abdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:13.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.364+0000 7f13977fe640 1 -- 172.21.15.117:0/4276070227 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f13980b4d00 msgr2=0x7f1398139190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:13.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.364+0000 7f13977fe640 1 --2- 172.21.15.117:0/4276070227 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f13980b4d00 0x7f1398139190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:13.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.364+0000 7f13977fe640 10 monclient: _finish_hunting 0 2024-05-15T12:43:13.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.364+0000 7f13977fe640 1 monclient: found mon.noname-a 2024-05-15T12:43:13.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.364+0000 7f13977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:13.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.364+0000 7f13977fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.117:3300/0 2024-05-15T12:43:13.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.364+0000 7f13977fe640 1 -- 172.21.15.117:0/4276070227 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13980ae130 con 0x7f13980ae830 2024-05-15T12:43:13.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.366+0000 7f13977fe640 1 --2- 172.21.15.117:0/4276070227 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13980ae830 0x7f13981396d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f139000b9e0 tx=0x7f139000bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:13.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.366+0000 7f138f7fe640 10 client.?.objecter ms_handle_connect 0x7f13980ae830 2024-05-15T12:43:13.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.366+0000 7f138f7fe640 10 client.?.objecter resend_mon_ops 2024-05-15T12:43:13.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.371+0000 7f138f7fe640 1 -- 172.21.15.117:0/4276070227 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1390019070 con 0x7f13980ae830 2024-05-15T12:43:13.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.371+0000 7f138f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:13.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.371+0000 7f138f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T12:43:13.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.371+0000 7f138f7fe640 10 monclient: dump: 2024-05-15T12:43:13.361 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:13.361 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:13.362 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.362 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.362 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:13.362 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:13.362 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:13.362 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:13.362 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:13.362 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:13.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.371+0000 7f138f7fe640 10 monclient: _finish_auth 0 2024-05-15T12:43:13.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.371+0000 7f138f7fe640 10 monclient: _check_auth_tickets 2024-05-15T12:43:13.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.371+0000 7f138f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:13.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.371+0000 7f138f7fe640 1 -- 172.21.15.117:0/4276070227 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f139000cac0 con 0x7f13980ae830 2024-05-15T12:43:13.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.371+0000 7f139d304640 5 monclient: authenticate success, global_id 4362 2024-05-15T12:43:13.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.371+0000 7f139d304640 10 monclient: _renew_subs 2024-05-15T12:43:13.363 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.371+0000 7f139d304640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:43:13.363 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.371+0000 7f139d304640 1 -- 172.21.15.117:0/4276070227 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13980ae410 con 0x7f13980ae830 2024-05-15T12:43:13.363 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.371+0000 7f138f7fe640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:13.363 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.371+0000 7f138f7fe640 1 -- 172.21.15.117:0/4276070227 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1390014440 con 0x7f13980ae830 2024-05-15T12:43:13.363 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.371+0000 7f138f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:13.363 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.371+0000 7f138f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T12:43:13.363 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.371+0000 7f138f7fe640 10 monclient: dump: 2024-05-15T12:43:13.363 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:13.363 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:13.363 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.363 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.363 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:13.363 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:13.363 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:13.363 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:13.364 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:13.364 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:13.364 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.371+0000 7f13957fa640 4 set_mon_vals no callback set 2024-05-15T12:43:13.364 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.371+0000 7f139d304640 10 client.4362.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:43:13.364 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.371+0000 7f139d304640 10 monclient: _renew_subs 2024-05-15T12:43:13.364 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.371+0000 7f139d304640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:43:13.364 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.371+0000 7f139d304640 1 -- 172.21.15.117:0/4276070227 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1398139c10 con 0x7f13980ae830 2024-05-15T12:43:13.364 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.371+0000 7f139d304640 1 librados: init done 2024-05-15T12:43:13.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.372+0000 7f139d304640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:13.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.372+0000 7f139d304640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:13.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.372+0000 7f139d304640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:43:13.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.372+0000 7f139d304640 1 -- 172.21.15.117:0/4276070227 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f139804f0e0 con 0x7f13980ae830 2024-05-15T12:43:13.377 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-05-15T12:43:13.387 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.397+0000 7f138f7fe640 1 -- 172.21.15.117:0/4276070227 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f139001c070 con 0x7f13980ae830 2024-05-15T12:43:13.387 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.397+0000 7f138f7fe640 1 --2- 172.21.15.117:0/4276070227 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f137806ce60 0x7f137806f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:13.388 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.398+0000 7f1396ffd640 1 --2- 172.21.15.117:0/4276070227 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f137806ce60 0x7f137806f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:13.388 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.398+0000 7f1396ffd640 10 monclient: get_auth_request con 0x7f137806ce60 auth_method 0 2024-05-15T12:43:13.388 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.398+0000 7f1396ffd640 1 --2- 172.21.15.117:0/4276070227 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f137806ce60 0x7f137806f320 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1388036000 tx=0x7f1388030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:13.388 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.398+0000 7f138f7fe640 10 client.4362.objecter ms_handle_connect 0x7f137806ce60 2024-05-15T12:43:13.391 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.401+0000 7f138f7fe640 1 -- 172.21.15.117:0/4276070227 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f1390056b50 con 0x7f13980ae830 2024-05-15T12:43:13.391 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.401+0000 7f138f7fe640 10 client.4362.objecter ms_dispatch 0x7f1398000b90 osd_map(26..26 src has 1..26) v4 2024-05-15T12:43:13.391 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.401+0000 7f138f7fe640 3 client.4362.objecter handle_osd_map got epochs [26,26] > 0 2024-05-15T12:43:13.391 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.401+0000 7f138f7fe640 3 client.4362.objecter handle_osd_map decoding full epoch 26 2024-05-15T12:43:13.391 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.401+0000 7f138f7fe640 20 client.4362.objecter dump_active .. 0 homeless 2024-05-15T12:43:13.429 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.439+0000 7f8a6affd640 1 -- 172.21.15.117:0/3102839906 <== mon.2 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f8a741d87d0 con 0x7f8a7406c900 2024-05-15T12:43:13.429 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.439+0000 7f8a6affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:13.429 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.439+0000 7f8a6affd640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:43:13.440 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.450+0000 7f2ec29a3640 10 monclient: get_monmap_and_config 2024-05-15T12:43:13.440 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.450+0000 7f2ec29a3640 10 monclient: build_initial_monmap 2024-05-15T12:43:13.440 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.450+0000 7f2ec29a3640 10 monclient: monmap: 2024-05-15T12:43:13.440 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:13.441 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:13.441 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:43:13.451614+0000 2024-05-15T12:43:13.441 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:43:13.451614+0000 2024-05-15T12:43:13.441 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:13.441 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:13.441 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T12:43:13.441 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T12:43:13.441 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T12:43:13.441 INFO:teuthology.orchestra.run.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T12:43:13.441 INFO:teuthology.orchestra.run.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T12:43:13.441 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:13.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.451+0000 7f2ec29a3640 1 Processor -- start 2024-05-15T12:43:13.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.451+0000 7f2ec29a3640 1 -- start start 2024-05-15T12:43:13.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.451+0000 7f2ec29a3640 10 monclient: init 2024-05-15T12:43:13.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.452+0000 7f2ec29a3640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:13.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.452+0000 7f2ec29a3640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-05-15T12:43:13.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.452+0000 7f2ec29a3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2ebc0b4d00 addr v1:172.21.15.178:6789/0 2024-05-15T12:43:13.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.452+0000 7f2ec29a3640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f2ebc0b50e0 0x7f2ebc0b1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:13.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.452+0000 7f2ec29a3640 10 monclient(hunting): picked mon.noname-a con 0x7f2ebc0b50e0 addr v2:172.21.15.117:3300/0 2024-05-15T12:43:13.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.452+0000 7f2ec29a3640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f2ebc0a94b0 0x7f2ebc0b0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:13.444 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.452+0000 7f2ec29a3640 10 monclient(hunting): picked mon.noname-c con 0x7f2ebc0a94b0 addr v2:172.21.15.178:3300/0 2024-05-15T12:43:13.444 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.452+0000 7f2ec29a3640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ebc0ab1b0 con 0x7f2ebc0b4d00 2024-05-15T12:43:13.444 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.452+0000 7f2ec29a3640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:13.444 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.452+0000 7f2ec29a3640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ebc0b23c0 con 0x7f2ebc0b50e0 2024-05-15T12:43:13.444 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.452+0000 7f2ec29a3640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:13.444 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.452+0000 7f2ec29a3640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ebc0b1520 con 0x7f2ebc0a94b0 2024-05-15T12:43:13.444 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.452+0000 7f2ec29a3640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:13.444 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.452+0000 7f2ec29a3640 10 monclient(hunting): authenticate will time out at 1356.694121s 2024-05-15T12:43:13.504 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.514+0000 7f35fd934640 10 monclient: get_monmap_and_config 2024-05-15T12:43:13.504 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.514+0000 7f35fd934640 10 monclient: build_initial_monmap 2024-05-15T12:43:13.504 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.514+0000 7f35fd934640 10 monclient: monmap: 2024-05-15T12:43:13.504 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:13.504 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:13.505 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:43:13.515245+0000 2024-05-15T12:43:13.505 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:43:13.515245+0000 2024-05-15T12:43:13.505 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:13.505 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:13.505 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T12:43:13.505 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T12:43:13.505 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T12:43:13.505 INFO:teuthology.orchestra.run.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T12:43:13.505 INFO:teuthology.orchestra.run.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T12:43:13.505 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:13.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.515+0000 7f35fd934640 1 Processor -- start 2024-05-15T12:43:13.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.515+0000 7f35fd934640 1 -- start start 2024-05-15T12:43:13.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.515+0000 7f35fd934640 10 monclient: init 2024-05-15T12:43:13.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.515+0000 7f35fd934640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:13.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.515+0000 7f35fd934640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-05-15T12:43:13.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.515+0000 7f35fd934640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f35f80b4d00 0x7f35f80b1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:13.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.515+0000 7f35fd934640 10 monclient(hunting): picked mon.noname-a con 0x7f35f80b4d00 addr v2:172.21.15.117:3300/0 2024-05-15T12:43:13.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.515+0000 7f35fd934640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f35f80b50e0 0x7f35f8145c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:13.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.515+0000 7f35fd934640 10 monclient(hunting): picked mon.noname-c con 0x7f35f80b50e0 addr v2:172.21.15.178:3300/0 2024-05-15T12:43:13.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.515+0000 7f35fd934640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f35f80a94b0 addr v1:172.21.15.178:6789/0 2024-05-15T12:43:13.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.515+0000 7f35fd934640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35f80ab1b0 con 0x7f35f80a94b0 2024-05-15T12:43:13.508 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.515+0000 7f35fd934640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:13.508 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.515+0000 7f35fd934640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35f80b23c0 con 0x7f35f80b4d00 2024-05-15T12:43:13.508 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.515+0000 7f35fd934640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:13.508 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.515+0000 7f35fd934640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35f80b16f0 con 0x7f35f80b50e0 2024-05-15T12:43:13.508 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.515+0000 7f35fd934640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:13.508 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.515+0000 7f35fd934640 10 monclient(hunting): authenticate will time out at 1356.757551s 2024-05-15T12:43:13.557 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-05-15T12:43:13.572 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.582+0000 7f2b4affd640 1 -- 172.21.15.117:0/794198670 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f2b50059180 con 0x7f2b5c0b86e0 2024-05-15T12:43:13.572 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.582+0000 7f2b4affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:13.572 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.582+0000 7f2b4affd640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:43:13.573 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.583+0000 7f35effff640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f35f80b50e0 0x7f35f8145c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:13.573 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.583+0000 7f2ec0f40640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f2ebc0a94b0 0x7f2ebc0b0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:13.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.584+0000 7f2ec0f40640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f2ebc0a94b0 0x7f2ebc0b0fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:42608/0 (socket says 172.21.15.117:42608) 2024-05-15T12:43:13.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.584+0000 7f2ec0f40640 1 -- 172.21.15.117:0/3397820270 learned_addr learned my addr 172.21.15.117:0/3397820270 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:43:13.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.584+0000 7f2ec0f40640 10 monclient(hunting): get_auth_request con 0x7f2ebc0a94b0 auth_method 0 2024-05-15T12:43:13.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.584+0000 7f2ebbfff640 1 --1- 172.21.15.117:0/3397820270 >> v1:172.21.15.178:6789/0 conn(0x7f2ebc0b4d00 0x7f2ebc145c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.117:53994/0 (socket says 172.21.15.117:53994) 2024-05-15T12:43:13.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.584+0000 7f2ec0f40640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:13.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.584+0000 7f2ec0f40640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:13.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.584+0000 7f2ec0f40640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:13.575 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.584+0000 7f35effff640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f35f80b50e0 0x7f35f8145c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:42610/0 (socket says 172.21.15.117:42610) 2024-05-15T12:43:13.575 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.584+0000 7f35effff640 1 -- 172.21.15.117:0/1063743979 learned_addr learned my addr 172.21.15.117:0/1063743979 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:43:13.575 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.584+0000 7f35effff640 10 monclient(hunting): get_auth_request con 0x7f35f80b50e0 auth_method 0 2024-05-15T12:43:13.575 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.584+0000 7f35effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:13.575 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.584+0000 7f35effff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:13.575 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.584+0000 7f35effff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:13.576 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.585+0000 7f2ec0f40640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:13.576 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.585+0000 7f2ec0f40640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:13.576 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.585+0000 7f2ec0f40640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:13.576 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.585+0000 7f35effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:13.576 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.585+0000 7f35effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:13.576 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.585+0000 7f35effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:13.577 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.587+0000 7f2ec0f40640 10 monclient(hunting): handle_auth_done global_id 4369 payload 274 2024-05-15T12:43:13.577 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.587+0000 7f2ec0f40640 1 -- 172.21.15.117:0/3397820270 >> v2:172.21.15.117:3300/0 conn(0x7f2ebc0b50e0 msgr2=0x7f2ebc0b1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:43:13.577 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.587+0000 7f2ec0f40640 1 --2- 172.21.15.117:0/3397820270 >> v2:172.21.15.117:3300/0 conn(0x7f2ebc0b50e0 0x7f2ebc0b1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:13.577 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.587+0000 7f2ec0f40640 1 -- 172.21.15.117:0/3397820270 >> v1:172.21.15.178:6789/0 conn(0x7f2ebc0b4d00 legacy=0x7f2ebc145c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:13.578 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.587+0000 7f2ec0f40640 10 monclient: _finish_hunting 0 2024-05-15T12:43:13.578 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.587+0000 7f2ec0f40640 1 monclient: found mon.noname-c 2024-05-15T12:43:13.578 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.587+0000 7f2ec0f40640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:13.578 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.587+0000 7f2ec0f40640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.178:3300/0 2024-05-15T12:43:13.578 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.587+0000 7f2ec0f40640 1 -- 172.21.15.117:0/3397820270 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ebc0b16d0 con 0x7f2ebc0a94b0 2024-05-15T12:43:13.580 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.590+0000 7f2ec0f40640 1 --2- 172.21.15.117:0/3397820270 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2ebc0a94b0 0x7f2ebc0b0fe0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc0aa590 tx=0x7f2eb402bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d387e89ab153b5a0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:13.582 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.592+0000 7f35effff640 10 monclient(hunting): handle_auth_done global_id 4372 payload 274 2024-05-15T12:43:13.583 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.592+0000 7f35effff640 1 -- 172.21.15.117:0/1063743979 >> v2:172.21.15.117:3300/0 conn(0x7f35f80b4d00 msgr2=0x7f35f80b1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:43:13.583 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.592+0000 7f35effff640 1 --2- 172.21.15.117:0/1063743979 >> v2:172.21.15.117:3300/0 conn(0x7f35f80b4d00 0x7f35f80b1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:13.583 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.592+0000 7f35effff640 1 -- 172.21.15.117:0/1063743979 >> v1:172.21.15.178:6789/0 conn(0x7f35f80a94b0 legacy=0x7f35f80b0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:43:13.583 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.592+0000 7f35effff640 10 monclient: _finish_hunting 0 2024-05-15T12:43:13.583 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.592+0000 7f35effff640 1 monclient: found mon.noname-c 2024-05-15T12:43:13.583 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.592+0000 7f35effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:13.583 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.592+0000 7f35effff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.178:3300/0 2024-05-15T12:43:13.583 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.592+0000 7f35effff640 1 -- 172.21.15.117:0/1063743979 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f8146160 con 0x7f35f80b50e0 2024-05-15T12:43:13.589 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.599+0000 7f35effff640 1 --2- 172.21.15.117:0/1063743979 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f35f80b50e0 0x7f35f8145c20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f35f8067a80 tx=0x7f35e002b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4185969a4cafa052 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:13.591 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.601+0000 7f2ebb7fe640 1 -- 172.21.15.117:0/3397820270 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb402ca80 con 0x7f2ebc0a94b0 2024-05-15T12:43:13.591 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.601+0000 7f2ebb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:13.591 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.601+0000 7f2ebb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:43:13.591 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.601+0000 7f2ebb7fe640 10 monclient: dump: 2024-05-15T12:43:13.591 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:13.591 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:13.591 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.591 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.592 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:13.592 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:13.592 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:13.592 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:13.592 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:13.592 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:13.592 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.601+0000 7f2ebb7fe640 10 monclient: _finish_auth 0 2024-05-15T12:43:13.592 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.601+0000 7f2ebb7fe640 10 monclient: _check_auth_tickets 2024-05-15T12:43:13.592 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.601+0000 7f2ebb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:13.592 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.601+0000 7f2ebb7fe640 1 -- 172.21.15.117:0/3397820270 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eb402cc20 con 0x7f2ebc0a94b0 2024-05-15T12:43:13.592 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.601+0000 7f2ec29a3640 5 monclient: authenticate success, global_id 4369 2024-05-15T12:43:13.592 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.601+0000 7f2ec29a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:43:13.592 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.601+0000 7f2ebb7fe640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:13.592 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.601+0000 7f2ec29a3640 10 monclient: get_monmap_and_config success 2024-05-15T12:43:13.593 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.601+0000 7f2ec29a3640 4 set_mon_vals no callback set 2024-05-15T12:43:13.593 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.601+0000 7f2ebb7fe640 1 -- 172.21.15.117:0/3397820270 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb40343f0 con 0x7f2ebc0a94b0 2024-05-15T12:43:13.593 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.601+0000 7f2ebb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:13.593 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.601+0000 7f2ebb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:43:13.593 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.601+0000 7f2ebb7fe640 10 monclient: dump: 2024-05-15T12:43:13.593 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:13.593 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:13.593 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.593 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.593 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:13.593 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:13.593 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:13.593 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:13.593 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:13.593 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:13.595 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.602+0000 7f2ec29a3640 10 monclient: shutdown 2024-05-15T12:43:13.595 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.602+0000 7f2ec29a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:13.595 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.602+0000 7f2ec29a3640 1 -- 172.21.15.117:0/3397820270 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2ebc0a94b0 msgr2=0x7f2ebc0b0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:13.595 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.602+0000 7f2ec29a3640 1 --2- 172.21.15.117:0/3397820270 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2ebc0a94b0 0x7f2ebc0b0fe0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc0aa590 tx=0x7f2eb402bee0 comp rx=0 tx=0).stop 2024-05-15T12:43:13.595 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.602+0000 7f2ec29a3640 1 -- 172.21.15.117:0/3397820270 shutdown_connections 2024-05-15T12:43:13.595 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.602+0000 7f2ec29a3640 1 --2- 172.21.15.117:0/3397820270 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2ebc0a94b0 0x7f2ebc0b0fe0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:13.595 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.602+0000 7f2ec29a3640 1 --2- 172.21.15.117:0/3397820270 >> v2:172.21.15.117:3300/0 conn(0x7f2ebc0b50e0 0x7f2ebc0b1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:13.596 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.602+0000 7f2ec29a3640 1 -- 172.21.15.117:0/3397820270 >> 172.21.15.117:0/3397820270 conn(0x7f2ebc066cc0 msgr2=0x7f2ebc0670d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:13.596 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.602+0000 7f2ec29a3640 1 -- 172.21.15.117:0/3397820270 shutdown_connections 2024-05-15T12:43:13.596 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.602+0000 7f2ec29a3640 1 -- 172.21.15.117:0/3397820270 wait complete. 2024-05-15T12:43:13.596 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.603+0000 7f2ec29a3640 10 monclient: build_initial_monmap 2024-05-15T12:43:13.596 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.603+0000 7f2ec29a3640 10 monclient: monmap: 2024-05-15T12:43:13.596 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:13.596 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:43:13.596 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-05-15T12:43:13.596 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-05-15T12:43:13.596 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:13.596 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:13.596 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T12:43:13.596 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T12:43:13.596 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T12:43:13.597 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:13.597 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.603+0000 7f2ec29a3640 1 librados: starting msgr at 2024-05-15T12:43:13.597 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.603+0000 7f2ec29a3640 1 librados: starting objecter 2024-05-15T12:43:13.597 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.604+0000 7f2ec29a3640 1 Processor -- start 2024-05-15T12:43:13.597 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.604+0000 7f2ec29a3640 1 -- start start 2024-05-15T12:43:13.597 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.604+0000 7f2ec29a3640 1 librados: setting wanted keys 2024-05-15T12:43:13.597 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.604+0000 7f2ec29a3640 1 librados: calling monclient init 2024-05-15T12:43:13.597 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.604+0000 7f2ec29a3640 10 monclient: init 2024-05-15T12:43:13.597 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.604+0000 7f35f67fc640 1 -- 172.21.15.117:0/1063743979 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e002e070 con 0x7f35f80b50e0 2024-05-15T12:43:13.598 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.604+0000 7f35f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:13.598 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.604+0000 7f35f67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:43:13.598 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.604+0000 7f35f67fc640 10 monclient: dump: 2024-05-15T12:43:13.598 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:13.598 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:13.598 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.598 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.598 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:13.598 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:13.598 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:13.598 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:13.598 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:13.598 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:13.598 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.604+0000 7f35f67fc640 10 monclient: _finish_auth 0 2024-05-15T12:43:13.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.604+0000 7f35f67fc640 10 monclient: _check_auth_tickets 2024-05-15T12:43:13.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.604+0000 7f35f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:13.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.604+0000 7f35f67fc640 1 -- 172.21.15.117:0/1063743979 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35e0037070 con 0x7f35f80b50e0 2024-05-15T12:43:13.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.604+0000 7f35fd934640 5 monclient: authenticate success, global_id 4372 2024-05-15T12:43:13.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.604+0000 7f35fd934640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:43:13.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.604+0000 7f35f67fc640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:13.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.604+0000 7f35f67fc640 1 -- 172.21.15.117:0/1063743979 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e0031a00 con 0x7f35f80b50e0 2024-05-15T12:43:13.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.604+0000 7f35f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:13.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.604+0000 7f35f67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:43:13.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.604+0000 7f35f67fc640 10 monclient: dump: 2024-05-15T12:43:13.599 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:13.599 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:13.599 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.599 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.600 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:13.600 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:13.600 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:13.600 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:13.600 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:13.600 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:13.600 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.604+0000 7f35fd934640 10 monclient: get_monmap_and_config success 2024-05-15T12:43:13.600 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.604+0000 7f35fd934640 4 set_mon_vals no callback set 2024-05-15T12:43:13.601 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.605+0000 7f2ec29a3640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:13.601 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.605+0000 7f2ec29a3640 10 monclient: _add_conns ranks=[1,0,2] 2024-05-15T12:43:13.601 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.605+0000 7f2ec29a3640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2ebc0a94b0 0x7f2ebc1422c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:13.601 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.605+0000 7f2ec29a3640 10 monclient(hunting): picked mon.noname-b con 0x7f2ebc0a94b0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:43:13.601 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.605+0000 7f2ec29a3640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2ebc0b4d00 0x7f2ebc13d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:13.601 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.605+0000 7f2ec29a3640 10 monclient(hunting): picked mon.noname-a con 0x7f2ebc0b4d00 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T12:43:13.601 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.605+0000 7f2ec29a3640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2ebc13c2a0 0x7f2ebc13e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:13.601 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.605+0000 7f2ec29a3640 10 monclient(hunting): picked mon.noname-c con 0x7f2ebc13c2a0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:43:13.602 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.605+0000 7f2ec29a3640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:13.602 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.605+0000 7f2ec29a3640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ebc14cfb0 con 0x7f2ebc0b4d00 2024-05-15T12:43:13.602 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.605+0000 7f2ec29a3640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:13.602 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.605+0000 7f2ec29a3640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ebc0a9b70 con 0x7f2ebc0a94b0 2024-05-15T12:43:13.602 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.605+0000 7f2ec29a3640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:13.602 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.605+0000 7f2ec29a3640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ebc14d130 con 0x7f2ebc13c2a0 2024-05-15T12:43:13.602 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.605+0000 7f2ec29a3640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:13.602 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.605+0000 7f2ec29a3640 10 monclient(hunting): authenticate will time out at 1356.847074s 2024-05-15T12:43:13.603 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.605+0000 7f35fd934640 10 monclient: shutdown 2024-05-15T12:43:13.603 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.605+0000 7f35fd934640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:13.603 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.605+0000 7f35fd934640 1 -- 172.21.15.117:0/1063743979 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f35f80b50e0 msgr2=0x7f35f8145c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:13.603 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.605+0000 7f35fd934640 1 --2- 172.21.15.117:0/1063743979 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f35f80b50e0 0x7f35f8145c20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f35f8067a80 tx=0x7f35e002b880 comp rx=0 tx=0).stop 2024-05-15T12:43:13.603 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.605+0000 7f35fd934640 1 -- 172.21.15.117:0/1063743979 shutdown_connections 2024-05-15T12:43:13.603 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.605+0000 7f35fd934640 1 --2- 172.21.15.117:0/1063743979 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f35f80b50e0 0x7f35f8145c20 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:13.603 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.605+0000 7f35fd934640 1 --2- 172.21.15.117:0/1063743979 >> v2:172.21.15.117:3300/0 conn(0x7f35f80b4d00 0x7f35f80b1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:13.603 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.605+0000 7f35fd934640 1 -- 172.21.15.117:0/1063743979 >> 172.21.15.117:0/1063743979 conn(0x7f35f8066cc0 msgr2=0x7f35f80670d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:13.603 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.606+0000 7f35fd934640 1 -- 172.21.15.117:0/1063743979 shutdown_connections 2024-05-15T12:43:13.604 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.606+0000 7f2ebbfff640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2ebc0a94b0 0x7f2ebc1422c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:13.604 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.606+0000 7f35fd934640 1 -- 172.21.15.117:0/1063743979 wait complete. 2024-05-15T12:43:13.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.606+0000 7f2ec0f40640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2ebc13c2a0 0x7f2ebc13e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:13.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.606+0000 7f2ec0f40640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2ebc13c2a0 0x7f2ebc13e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.117:49108/0 (socket says 172.21.15.117:49108) 2024-05-15T12:43:13.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.606+0000 7f2ec0f40640 1 -- 172.21.15.117:0/1684657204 learned_addr learned my addr 172.21.15.117:0/1684657204 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:43:13.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.606+0000 7f2ec0f40640 10 monclient(hunting): get_auth_request con 0x7f2ebc13c2a0 auth_method 0 2024-05-15T12:43:13.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.606+0000 7f2ec0f40640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:13.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.606+0000 7f2ec0f40640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:13.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.606+0000 7f2ec0f40640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:13.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.606+0000 7f2ebbfff640 10 monclient(hunting): get_auth_request con 0x7f2ebc0a94b0 auth_method 0 2024-05-15T12:43:13.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.606+0000 7f2ebbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:13.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.606+0000 7f2ebbfff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:13.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.606+0000 7f2ebbfff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:13.606 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.607+0000 7f35fd934640 10 monclient: build_initial_monmap 2024-05-15T12:43:13.606 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.607+0000 7f35fd934640 10 monclient: monmap: 2024-05-15T12:43:13.606 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:13.606 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:43:13.606 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-05-15T12:43:13.606 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-05-15T12:43:13.606 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:13.606 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:13.606 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T12:43:13.607 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T12:43:13.607 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T12:43:13.607 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:13.607 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.607+0000 7f2ec0f40640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:13.607 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.607+0000 7f2ec0f40640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:13.607 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.607+0000 7f2ec0f40640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:13.607 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.607+0000 7f35fd934640 1 librados: starting msgr at 2024-05-15T12:43:13.608 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.607+0000 7f35fd934640 1 librados: starting objecter 2024-05-15T12:43:13.608 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.607+0000 7f2ebbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:13.608 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.607+0000 7f2ebbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:13.608 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.607+0000 7f2ebbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:13.610 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.608+0000 7f35fd934640 1 Processor -- start 2024-05-15T12:43:13.610 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.608+0000 7f35fd934640 1 -- start start 2024-05-15T12:43:13.610 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.608+0000 7f35fd934640 1 librados: setting wanted keys 2024-05-15T12:43:13.611 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.608+0000 7f35fd934640 1 librados: calling monclient init 2024-05-15T12:43:13.611 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.608+0000 7f35fd934640 10 monclient: init 2024-05-15T12:43:13.611 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.609+0000 7f35fd934640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:13.611 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.609+0000 7f35fd934640 10 monclient: _add_conns ranks=[0,1,2] 2024-05-15T12:43:13.611 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.609+0000 7f35fd934640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f35f80a94b0 0x7f35f81427a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:13.611 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.609+0000 7f35fd934640 10 monclient(hunting): picked mon.noname-a con 0x7f35f80a94b0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T12:43:13.611 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.609+0000 7f35fd934640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f35f80b4d00 0x7f35f813e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:13.611 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.609+0000 7f35fd934640 10 monclient(hunting): picked mon.noname-b con 0x7f35f80b4d00 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:43:13.611 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.609+0000 7f35fd934640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f35f813c2a0 0x7f35f813d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:13.611 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.609+0000 7f35fd934640 10 monclient(hunting): picked mon.noname-c con 0x7f35f813c2a0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:43:13.611 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.609+0000 7f35fd934640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:13.611 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.609+0000 7f35fd934640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f814cfb0 con 0x7f35f80a94b0 2024-05-15T12:43:13.611 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.609+0000 7f35fd934640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:13.611 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.609+0000 7f35fd934640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f80a9b70 con 0x7f35f80b4d00 2024-05-15T12:43:13.611 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.610+0000 7f35fd934640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:13.612 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.610+0000 7f35fd934640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35f814d130 con 0x7f35f813c2a0 2024-05-15T12:43:13.612 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.610+0000 7f35fd934640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:13.612 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.610+0000 7f35fd934640 10 monclient(hunting): authenticate will time out at 1356.852052s 2024-05-15T12:43:13.612 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.611+0000 7f2ebbfff640 10 monclient(hunting): handle_auth_done global_id 4375 payload 824 2024-05-15T12:43:13.612 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.611+0000 7f2ebbfff640 1 -- 172.21.15.117:0/1684657204 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2ebc13c2a0 msgr2=0x7f2ebc13e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:13.612 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.611+0000 7f2ebbfff640 1 --2- 172.21.15.117:0/1684657204 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2ebc13c2a0 0x7f2ebc13e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:13.612 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.611+0000 7f2ebbfff640 1 -- 172.21.15.117:0/1684657204 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2ebc0b4d00 msgr2=0x7f2ebc13d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:13.613 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.611+0000 7f2ebbfff640 1 --2- 172.21.15.117:0/1684657204 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2ebc0b4d00 0x7f2ebc13d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:13.613 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.611+0000 7f2ebbfff640 10 monclient: _finish_hunting 0 2024-05-15T12:43:13.613 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.611+0000 7f2ebbfff640 1 monclient: found mon.noname-b 2024-05-15T12:43:13.613 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.611+0000 7f2ebbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:13.613 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.611+0000 7f2ebbfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2024-05-15T12:43:13.613 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.611+0000 7f2ebbfff640 1 -- 172.21.15.117:0/1684657204 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ebc142800 con 0x7f2ebc0a94b0 2024-05-15T12:43:13.614 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.616+0000 7f35f77fe640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f35f813c2a0 0x7f35f813d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:13.614 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.616+0000 7f35f77fe640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f35f813c2a0 0x7f35f813d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.117:49124/0 (socket says 172.21.15.117:49124) 2024-05-15T12:43:13.614 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.616+0000 7f35f77fe640 1 -- 172.21.15.117:0/3546014827 learned_addr learned my addr 172.21.15.117:0/3546014827 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:43:13.614 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.616+0000 7f35f77fe640 10 monclient(hunting): get_auth_request con 0x7f35f813c2a0 auth_method 0 2024-05-15T12:43:13.614 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.616+0000 7f35f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:13.614 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.616+0000 7f35f77fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:13.614 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.616+0000 7f35f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:13.614 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.617+0000 7f35f77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:13.614 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.617+0000 7f35f77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:13.614 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.617+0000 7f35f77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:13.614 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.621+0000 7f35f77fe640 10 monclient(hunting): handle_auth_done global_id 4316 payload 824 2024-05-15T12:43:13.615 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.621+0000 7f35f77fe640 1 -- 172.21.15.117:0/3546014827 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f35f80b4d00 msgr2=0x7f35f813e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:13.615 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.621+0000 7f35f77fe640 1 --2- 172.21.15.117:0/3546014827 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f35f80b4d00 0x7f35f813e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:13.615 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.621+0000 7f35f77fe640 1 -- 172.21.15.117:0/3546014827 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f35f80a94b0 msgr2=0x7f35f81427a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:13.615 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.621+0000 7f35f77fe640 1 --2- 172.21.15.117:0/3546014827 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f35f80a94b0 0x7f35f81427a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:13.615 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.621+0000 7f35f77fe640 10 monclient: _finish_hunting 0 2024-05-15T12:43:13.615 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.621+0000 7f35f77fe640 1 monclient: found mon.noname-c 2024-05-15T12:43:13.615 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.621+0000 7f35f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:13.615 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.621+0000 7f35f77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.117:3301/0 2024-05-15T12:43:13.615 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.621+0000 7f35f77fe640 1 -- 172.21.15.117:0/3546014827 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f80ac2b0 con 0x7f35f813c2a0 2024-05-15T12:43:13.615 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.625+0000 7f35f77fe640 1 --2- 172.21.15.117:0/3546014827 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f35f813c2a0 0x7f35f813d8c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f35f000b9e0 tx=0x7f35f000bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:13.616 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.625+0000 7f35f4ff9640 10 client.?.objecter ms_handle_connect 0x7f35f813c2a0 2024-05-15T12:43:13.616 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.625+0000 7f35f4ff9640 10 client.?.objecter resend_mon_ops 2024-05-15T12:43:13.620 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.630+0000 7f2ebbfff640 1 --2- 172.21.15.117:0/1684657204 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2ebc0a94b0 0x7f2ebc1422c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2eac000f60 tx=0x7f2eac00c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:13.620 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.630+0000 7f2eb97fa640 10 client.?.objecter ms_handle_connect 0x7f2ebc0a94b0 2024-05-15T12:43:13.620 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.630+0000 7f2eb97fa640 10 client.?.objecter resend_mon_ops 2024-05-15T12:43:13.621 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.631+0000 7f35f4ff9640 1 -- 172.21.15.117:0/3546014827 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f0003b70 con 0x7f35f813c2a0 2024-05-15T12:43:13.621 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.631+0000 7f35f4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:13.621 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.631+0000 7f35f4ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:43:13.621 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.631+0000 7f35f4ff9640 10 monclient: dump: 2024-05-15T12:43:13.621 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:13.621 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:13.621 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.621 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.621 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:13.622 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:13.622 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:13.622 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:13.622 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:13.622 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:13.622 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.631+0000 7f35f4ff9640 10 monclient: _finish_auth 0 2024-05-15T12:43:13.622 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.631+0000 7f35f4ff9640 10 monclient: _check_auth_tickets 2024-05-15T12:43:13.622 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.631+0000 7f35f4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:13.622 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.631+0000 7f35f4ff9640 1 -- 172.21.15.117:0/3546014827 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35f0003d10 con 0x7f35f813c2a0 2024-05-15T12:43:13.622 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.631+0000 7f35fd934640 5 monclient: authenticate success, global_id 4316 2024-05-15T12:43:13.622 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.631+0000 7f35fd934640 10 monclient: _renew_subs 2024-05-15T12:43:13.622 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.631+0000 7f35fd934640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T12:43:13.622 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.631+0000 7f35fd934640 1 -- 172.21.15.117:0/3546014827 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35f813e2d0 con 0x7f35f813c2a0 2024-05-15T12:43:13.622 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.631+0000 7f35f4ff9640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:13.623 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.631+0000 7f35f4ff9640 1 -- 172.21.15.117:0/3546014827 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f000cae0 con 0x7f35f813c2a0 2024-05-15T12:43:13.623 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.631+0000 7f35f4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:13.623 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.631+0000 7f35fd934640 10 client.4316.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:43:13.623 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.631+0000 7f35f4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T12:43:13.623 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.631+0000 7f35f4ff9640 10 monclient: dump: 2024-05-15T12:43:13.623 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:13.623 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:13.623 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.623 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.623 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:13.623 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:13.623 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:13.623 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:13.623 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:13.623 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:13.624 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.631+0000 7f35fd934640 10 monclient: _renew_subs 2024-05-15T12:43:13.624 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.631+0000 7f35fd934640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T12:43:13.624 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.631+0000 7f35fd934640 1 -- 172.21.15.117:0/3546014827 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35f813de00 con 0x7f35f813c2a0 2024-05-15T12:43:13.624 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.631+0000 7f35fd934640 1 librados: init done 2024-05-15T12:43:13.624 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.631+0000 7f35f5ffb640 4 set_mon_vals no callback set 2024-05-15T12:43:13.624 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.632+0000 7f35fd934640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:13.624 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.632+0000 7f35fd934640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:13.624 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.632+0000 7f35fd934640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T12:43:13.624 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.632+0000 7f35fd934640 1 -- 172.21.15.117:0/3546014827 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35f80b5b40 con 0x7f35f813c2a0 2024-05-15T12:43:13.626 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.636+0000 7f2eb97fa640 1 -- 172.21.15.117:0/1684657204 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eac0074e0 con 0x7f2ebc0a94b0 2024-05-15T12:43:13.626 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.636+0000 7f2eb97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:13.626 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.636+0000 7f2eb97fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:43:13.626 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.636+0000 7f2eb97fa640 10 monclient: dump: 2024-05-15T12:43:13.627 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:13.627 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:13.627 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.627 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.627 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:13.627 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:13.627 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:13.627 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:13.627 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:13.627 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:13.627 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.636+0000 7f2eb97fa640 10 monclient: _finish_auth 0 2024-05-15T12:43:13.627 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.636+0000 7f2eb97fa640 10 monclient: _check_auth_tickets 2024-05-15T12:43:13.627 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.636+0000 7f2eb97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:13.627 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.636+0000 7f2eb97fa640 1 -- 172.21.15.117:0/1684657204 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eac0047e0 con 0x7f2ebc0a94b0 2024-05-15T12:43:13.628 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.636+0000 7f2eb97fa640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:13.628 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.636+0000 7f2ebaffd640 4 set_mon_vals no callback set 2024-05-15T12:43:13.629 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.637+0000 7f2ec29a3640 5 monclient: authenticate success, global_id 4375 2024-05-15T12:43:13.629 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.637+0000 7f2ec29a3640 10 monclient: _renew_subs 2024-05-15T12:43:13.629 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.637+0000 7f2ec29a3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:43:13.629 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.637+0000 7f2ec29a3640 1 -- 172.21.15.117:0/1684657204 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ebc0ac0b0 con 0x7f2ebc0a94b0 2024-05-15T12:43:13.629 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.637+0000 7f2eb97fa640 1 -- 172.21.15.117:0/1684657204 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eac004bd0 con 0x7f2ebc0a94b0 2024-05-15T12:43:13.629 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.637+0000 7f2eb97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:13.629 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.637+0000 7f2eb97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:43:13.629 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.637+0000 7f2eb97fa640 10 monclient: dump: 2024-05-15T12:43:13.629 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:13.629 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:13.629 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.630 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.630 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:13.630 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:13.630 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:13.630 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:13.630 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:13.630 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:13.630 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.637+0000 7f2ec29a3640 10 client.4375.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:43:13.630 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.637+0000 7f2ec29a3640 10 monclient: _renew_subs 2024-05-15T12:43:13.630 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.637+0000 7f2ec29a3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:43:13.630 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.637+0000 7f2ec29a3640 1 -- 172.21.15.117:0/1684657204 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ebc13eac0 con 0x7f2ebc0a94b0 2024-05-15T12:43:13.630 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.637+0000 7f2ec29a3640 1 librados: init done 2024-05-15T12:43:13.630 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.638+0000 7f2ec29a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:13.630 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.639+0000 7f2ec29a3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:13.630 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.639+0000 7f2ec29a3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:43:13.631 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.639+0000 7f2ec29a3640 1 -- 172.21.15.117:0/1684657204 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ebc04f0e0 con 0x7f2ebc0a94b0 2024-05-15T12:43:13.631 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-05-15T12:43:13.661 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.671+0000 7f8a7c5aa640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-05-15T12:43:13.661 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.671+0000 7f8a7c5aa640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-05-15T12:43:13.661 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.671+0000 7f8a7c5aa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T12:43:13.662 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.671+0000 7f35f4ff9640 1 -- 172.21.15.117:0/3546014827 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f35f000cc80 con 0x7f35f813c2a0 2024-05-15T12:43:13.662 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.671+0000 7f8a7c5aa640 1 -- 172.21.15.117:0/3102839906 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f8a74143e20 con 0x7f8a7406c900 2024-05-15T12:43:13.663 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.671+0000 7f35f4ff9640 1 --2- 172.21.15.117:0/3546014827 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f35d806ce60 0x7f35d806f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:13.663 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.672+0000 7f35f6ffd640 1 --2- 172.21.15.117:0/3546014827 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f35d806ce60 0x7f35d806f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:13.663 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.672+0000 7f35f6ffd640 10 monclient: get_auth_request con 0x7f35d806ce60 auth_method 0 2024-05-15T12:43:13.663 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.673+0000 7f35f4ff9640 10 client.4316.objecter ms_handle_connect 0x7f35d806ce60 2024-05-15T12:43:13.663 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.673+0000 7f35f6ffd640 1 --2- 172.21.15.117:0/3546014827 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f35d806ce60 0x7f35d806f320 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f35e8009710 tx=0x7f35e8006c90 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:13.676 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.686+0000 7f5b69f5d640 10 monclient: get_monmap_and_config 2024-05-15T12:43:13.676 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.686+0000 7f5b69f5d640 10 monclient: build_initial_monmap 2024-05-15T12:43:13.676 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.686+0000 7f5b69f5d640 10 monclient: monmap: 2024-05-15T12:43:13.676 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:13.676 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:13.676 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:43:13.687120+0000 2024-05-15T12:43:13.676 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:43:13.687120+0000 2024-05-15T12:43:13.676 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:13.676 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:13.676 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T12:43:13.676 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T12:43:13.677 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T12:43:13.677 INFO:teuthology.orchestra.run.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T12:43:13.677 INFO:teuthology.orchestra.run.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T12:43:13.677 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:13.678 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.686+0000 7f5b69f5d640 1 Processor -- start 2024-05-15T12:43:13.678 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.686+0000 7f5b69f5d640 1 -- start start 2024-05-15T12:43:13.678 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.686+0000 7f5b69f5d640 10 monclient: init 2024-05-15T12:43:13.678 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.687+0000 7f5b69f5d640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:13.678 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.687+0000 7f5b69f5d640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-05-15T12:43:13.678 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.687+0000 7f5b69f5d640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5b640b4d00 0x7f5b640b1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:13.678 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.687+0000 7f5b69f5d640 10 monclient(hunting): picked mon.noname-b con 0x7f5b640b4d00 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:43:13.678 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.687+0000 7f5b69f5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5b640b50e0 addr v1:172.21.15.117:6789/0 2024-05-15T12:43:13.678 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.687+0000 7f5b69f5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5b640a94b0 addr v1:172.21.15.178:6789/0 2024-05-15T12:43:13.679 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.687+0000 7f5b69f5d640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b640ab1b0 con 0x7f5b640b50e0 2024-05-15T12:43:13.679 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.687+0000 7f5b69f5d640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b640a9940 con 0x7f5b640a94b0 2024-05-15T12:43:13.679 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.687+0000 7f5b69f5d640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:13.679 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.687+0000 7f5b69f5d640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b640b23c0 con 0x7f5b640b4d00 2024-05-15T12:43:13.679 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.687+0000 7f5b69f5d640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:13.679 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.687+0000 7f5b69f5d640 10 monclient(hunting): authenticate will time out at 1356.929566s 2024-05-15T12:43:13.691 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.701+0000 7f138f7fe640 1 -- 172.21.15.117:0/4276070227 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f1390023020 con 0x7f13980ae830 2024-05-15T12:43:13.691 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.701+0000 7f138f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:13.691 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.701+0000 7f138f7fe640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:43:13.709 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.719+0000 7f5b62ffd640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f5b640b50e0 0x7f5b64145c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:48736/0 (socket says 172.21.15.117:48736) 2024-05-15T12:43:13.709 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.719+0000 7f5b62ffd640 1 -- 172.21.15.117:0/3638502352 learned_addr learned my addr 172.21.15.117:0/3638502352 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-05-15T12:43:13.716 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.726+0000 7f5b627fc640 1 -- 172.21.15.117:0/3638502352 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2588851277 0 0) 0x7f5b640ab1b0 con 0x7f5b640b50e0 2024-05-15T12:43:13.716 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.726+0000 7f5b627fc640 10 monclient(hunting): _negotiate 2024-05-15T12:43:13.716 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.726+0000 7f5b627fc640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:13.716 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.726+0000 7f5b627fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:13.717 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.726+0000 7f5b627fc640 10 monclient(hunting): my global_id is 4365 2024-05-15T12:43:13.717 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.726+0000 7f5b627fc640 1 -- 172.21.15.117:0/3638502352 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b4c003520 con 0x7f5b640b50e0 2024-05-15T12:43:13.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.730+0000 7f5b627fc640 1 -- 172.21.15.117:0/3638502352 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4056101775 0 0) 0x7f5b4c003520 con 0x7f5b640b50e0 2024-05-15T12:43:13.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.730+0000 7f5b627fc640 1 -- 172.21.15.117:0/3638502352 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5b640b4d00 msgr2=0x7f5b640b1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:43:13.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.730+0000 7f5b627fc640 1 --2- 172.21.15.117:0/3638502352 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5b640b4d00 0x7f5b640b1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:13.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.730+0000 7f5b627fc640 1 -- 172.21.15.117:0/3638502352 >> v1:172.21.15.178:6789/0 conn(0x7f5b640a94b0 legacy=0x7f5b640b0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:43:13.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.730+0000 7f5b627fc640 10 monclient: _finish_hunting 0 2024-05-15T12:43:13.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.730+0000 7f5b627fc640 1 monclient: found mon.noname-a-legacy 2024-05-15T12:43:13.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.730+0000 7f5b627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:13.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.730+0000 7f5b627fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.117:6789/0 2024-05-15T12:43:13.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.730+0000 7f5b627fc640 1 -- 172.21.15.117:0/3638502352 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b640a9b10 con 0x7f5b640b50e0 2024-05-15T12:43:13.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.730+0000 7f5b627fc640 10 monclient: _finish_auth 0 2024-05-15T12:43:13.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.730+0000 7f5b627fc640 10 monclient: _check_auth_tickets 2024-05-15T12:43:13.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.730+0000 7f5b627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:13.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.730+0000 7f5b627fc640 1 -- 172.21.15.117:0/3638502352 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3166367242 0 0) 0x7f5b58001670 con 0x7f5b640b50e0 2024-05-15T12:43:13.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.730+0000 7f5b627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:13.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.730+0000 7f5b627fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-05-15T12:43:13.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.730+0000 7f5b627fc640 10 monclient: dump: 2024-05-15T12:43:13.722 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:13.722 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:13.722 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.722 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.722 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:13.722 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:13.722 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:13.722 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:13.722 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:13.722 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:13.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.730+0000 7f5b627fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] but i'm connected to v1:172.21.15.117:6789/0, reconnecting 2024-05-15T12:43:13.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.730+0000 7f5b627fc640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:13.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.730+0000 7f5b627fc640 1 -- 172.21.15.117:0/3638502352 >> v1:172.21.15.117:6789/0 conn(0x7f5b640b50e0 legacy=0x7f5b64145c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:13.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.730+0000 7f5b627fc640 10 monclient: _add_conns ranks=[1,0,2] 2024-05-15T12:43:13.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.730+0000 7f5b627fc640 1 --2- 172.21.15.117:0/3638502352 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5b4c0039e0 0x7f5b4c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:13.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.730+0000 7f5b627fc640 10 monclient(hunting): picked mon.b con 0x7f5b4c0039e0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:43:13.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.730+0000 7f5b627fc640 1 --2- 172.21.15.117:0/3638502352 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5b4c004800 0x7f5b4c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:13.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.730+0000 7f5b627fc640 10 monclient(hunting): picked mon.a con 0x7f5b4c004800 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T12:43:13.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.730+0000 7f5b627fc640 1 --2- 172.21.15.117:0/3638502352 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5b640b4d00 0x7f5b4c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:13.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.730+0000 7f5b627fc640 10 monclient(hunting): picked mon.c con 0x7f5b640b4d00 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:43:13.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.730+0000 7f5b627fc640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:13.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.730+0000 7f5b627fc640 1 -- 172.21.15.117:0/3638502352 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b640b23c0 con 0x7f5b4c004800 2024-05-15T12:43:13.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.730+0000 7f5b627fc640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:13.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.730+0000 7f5b627fc640 1 -- 172.21.15.117:0/3638502352 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b4c005600 con 0x7f5b4c0039e0 2024-05-15T12:43:13.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.730+0000 7f5b627fc640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:13.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.730+0000 7f5b627fc640 1 -- 172.21.15.117:0/3638502352 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b4c00bf00 con 0x7f5b640b4d00 2024-05-15T12:43:13.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.730+0000 7f5b627fc640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:13.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.730+0000 7f5b627fc640 10 monclient(hunting): _finish_auth 0 2024-05-15T12:43:13.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.732+0000 7f5b62ffd640 1 --2- 172.21.15.117:0/3638502352 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5b4c004800 0x7f5b4c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:13.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.732+0000 7f5b62ffd640 10 monclient(hunting): get_auth_request con 0x7f5b4c004800 auth_method 0 2024-05-15T12:43:13.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.732+0000 7f5b62ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:13.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.732+0000 7f5b62ffd640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:13.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.732+0000 7f5b62ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-05-15T12:43:13.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.733+0000 7f5b62ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:13.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.733+0000 7f5b62ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:13.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.733+0000 7f5b62ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-05-15T12:43:13.726 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.736+0000 7f5b62ffd640 10 monclient(hunting): handle_auth_done global_id 4365 payload 293 2024-05-15T12:43:13.726 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.736+0000 7f5b62ffd640 1 -- 172.21.15.117:0/3638502352 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5b640b4d00 msgr2=0x7f5b4c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:43:13.726 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.736+0000 7f5b62ffd640 1 --2- 172.21.15.117:0/3638502352 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5b640b4d00 0x7f5b4c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:13.726 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.736+0000 7f5b62ffd640 1 -- 172.21.15.117:0/3638502352 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5b4c0039e0 msgr2=0x7f5b4c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:43:13.726 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.736+0000 7f5b62ffd640 1 --2- 172.21.15.117:0/3638502352 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5b4c0039e0 0x7f5b4c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:13.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.736+0000 7f5b62ffd640 10 monclient: _finish_hunting 0 2024-05-15T12:43:13.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.736+0000 7f5b62ffd640 1 monclient: found mon.a 2024-05-15T12:43:13.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.736+0000 7f5b62ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:13.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.736+0000 7f5b62ffd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:43:13.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.736+0000 7f5b62ffd640 1 -- 172.21.15.117:0/3638502352 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b640a9b10 con 0x7f5b4c004800 2024-05-15T12:43:13.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.736+0000 7f5b62ffd640 10 monclient: _finish_auth 0 2024-05-15T12:43:13.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.736+0000 7f5b62ffd640 10 monclient: _check_auth_tickets 2024-05-15T12:43:13.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.736+0000 7f5b62ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:13.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.736+0000 7f5b69f5d640 5 monclient: authenticate success, global_id 4365 2024-05-15T12:43:13.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.736+0000 7f5b69f5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:43:13.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.739+0000 7f5b62ffd640 1 --2- 172.21.15.117:0/3638502352 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5b4c004800 0x7f5b4c004c80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5b5800c290 tx=0x7f5b4c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6ba4e7768070d6 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:13.733 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.743+0000 7f5b627fc640 1 -- 172.21.15.117:0/3638502352 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b58007d50 con 0x7f5b4c004800 2024-05-15T12:43:13.733 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.743+0000 7f5b627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:13.734 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.743+0000 7f5b627fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T12:43:13.734 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.743+0000 7f5b627fc640 10 monclient: dump: 2024-05-15T12:43:13.734 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:13.734 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:13.734 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.734 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.734 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:13.734 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:13.734 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:13.734 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:13.734 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:13.734 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:13.734 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.743+0000 7f5b627fc640 1 -- 172.21.15.117:0/3638502352 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b58030070 con 0x7f5b4c004800 2024-05-15T12:43:13.734 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.743+0000 7f5b627fc640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:13.735 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.743+0000 7f5b627fc640 1 -- 172.21.15.117:0/3638502352 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b5802ce20 con 0x7f5b4c004800 2024-05-15T12:43:13.735 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.743+0000 7f5b627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:13.735 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.743+0000 7f5b627fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T12:43:13.735 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.743+0000 7f5b627fc640 10 monclient: dump: 2024-05-15T12:43:13.735 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:13.735 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:13.735 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.735 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.735 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:13.735 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:13.735 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:13.735 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:13.735 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:13.735 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:13.735 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.743+0000 7f5b69f5d640 10 monclient: get_monmap_and_config success 2024-05-15T12:43:13.736 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.743+0000 7f5b69f5d640 4 set_mon_vals no callback set 2024-05-15T12:43:13.738 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.744+0000 7f5b69f5d640 10 monclient: shutdown 2024-05-15T12:43:13.738 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.744+0000 7f5b69f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:13.738 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.744+0000 7f5b69f5d640 1 -- 172.21.15.117:0/3638502352 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5b4c004800 msgr2=0x7f5b4c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:13.738 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.744+0000 7f5b69f5d640 1 --2- 172.21.15.117:0/3638502352 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5b4c004800 0x7f5b4c004c80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5b5800c290 tx=0x7f5b4c005830 comp rx=0 tx=0).stop 2024-05-15T12:43:13.738 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.744+0000 7f5b63fff640 1 -- 172.21.15.117:0/3638502352 reap_dead start 2024-05-15T12:43:13.738 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.744+0000 7f5b69f5d640 1 -- 172.21.15.117:0/3638502352 shutdown_connections 2024-05-15T12:43:13.738 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.744+0000 7f5b69f5d640 1 -- 172.21.15.117:0/3638502352 >> 172.21.15.117:0/3638502352 conn(0x7f5b64066cc0 msgr2=0x7f5b640670d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:13.738 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.744+0000 7f5b69f5d640 1 -- 172.21.15.117:0/3638502352 shutdown_connections 2024-05-15T12:43:13.738 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.744+0000 7f5b69f5d640 1 -- 172.21.15.117:0/3638502352 wait complete. 2024-05-15T12:43:13.739 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.745+0000 7f5b69f5d640 10 monclient: build_initial_monmap 2024-05-15T12:43:13.739 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.745+0000 7f5b69f5d640 10 monclient: monmap: 2024-05-15T12:43:13.739 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:13.739 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:43:13.739 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-05-15T12:43:13.739 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-05-15T12:43:13.739 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:13.739 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:13.739 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T12:43:13.739 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T12:43:13.739 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T12:43:13.739 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:13.739 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.745+0000 7f5b69f5d640 1 librados: starting msgr at 2024-05-15T12:43:13.739 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.745+0000 7f5b69f5d640 1 librados: starting objecter 2024-05-15T12:43:13.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.745+0000 7f5b69f5d640 1 Processor -- start 2024-05-15T12:43:13.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.746+0000 7f5b69f5d640 1 -- start start 2024-05-15T12:43:13.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.746+0000 7f5b69f5d640 1 librados: setting wanted keys 2024-05-15T12:43:13.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.746+0000 7f5b69f5d640 1 librados: calling monclient init 2024-05-15T12:43:13.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.746+0000 7f5b69f5d640 10 monclient: init 2024-05-15T12:43:13.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.746+0000 7f5b69f5d640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:13.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.746+0000 7f5b69f5d640 10 monclient: _add_conns ranks=[2,1,0] 2024-05-15T12:43:13.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.746+0000 7f5b69f5d640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5b640aeb80 0x7f5b640af110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:13.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.746+0000 7f5b69f5d640 10 monclient(hunting): picked mon.noname-c con 0x7f5b640aeb80 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:43:13.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.746+0000 7f5b69f5d640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5b6413fa30 0x7f5b6413e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:13.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.746+0000 7f5b69f5d640 10 monclient(hunting): picked mon.noname-b con 0x7f5b6413fa30 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:43:13.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.746+0000 7f5b69f5d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5b6413eda0 0x7f5b641daf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:13.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.746+0000 7f5b69f5d640 10 monclient(hunting): picked mon.noname-a con 0x7f5b6413eda0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T12:43:13.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.746+0000 7f5b69f5d640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:13.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.746+0000 7f5b69f5d640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b640b16f0 con 0x7f5b6413eda0 2024-05-15T12:43:13.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.746+0000 7f5b69f5d640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:13.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.746+0000 7f5b69f5d640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b640b2110 con 0x7f5b6413fa30 2024-05-15T12:43:13.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.746+0000 7f5b69f5d640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:13.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.746+0000 7f5b69f5d640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b64146f80 con 0x7f5b640aeb80 2024-05-15T12:43:13.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.746+0000 7f5b69f5d640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:13.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.746+0000 7f5b69f5d640 10 monclient(hunting): authenticate will time out at 1356.988689s 2024-05-15T12:43:13.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.747+0000 7f5b63fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5b6413eda0 0x7f5b641daf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:13.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.748+0000 7f5b63fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5b6413eda0 0x7f5b641daf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:60400/0 (socket says 172.21.15.117:60400) 2024-05-15T12:43:13.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.748+0000 7f5b63fff640 1 -- 172.21.15.117:0/1553769346 learned_addr learned my addr 172.21.15.117:0/1553769346 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:43:13.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.748+0000 7f5b63fff640 10 monclient(hunting): get_auth_request con 0x7f5b6413eda0 auth_method 0 2024-05-15T12:43:13.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.748+0000 7f5b63fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:13.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.748+0000 7f5b63fff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:13.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.748+0000 7f5b63fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:13.742 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.748+0000 7f5b63fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:13.742 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.748+0000 7f5b63fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:13.742 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.748+0000 7f5b63fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:13.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.753+0000 7f5b63fff640 10 monclient(hunting): handle_auth_done global_id 4368 payload 824 2024-05-15T12:43:13.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.753+0000 7f5b63fff640 1 -- 172.21.15.117:0/1553769346 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5b640aeb80 msgr2=0x7f5b640af110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:13.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.753+0000 7f5b63fff640 1 --2- 172.21.15.117:0/1553769346 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5b640aeb80 0x7f5b640af110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:13.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.753+0000 7f5b63fff640 1 -- 172.21.15.117:0/1553769346 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5b6413fa30 msgr2=0x7f5b6413e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:13.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.753+0000 7f5b63fff640 1 --2- 172.21.15.117:0/1553769346 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5b6413fa30 0x7f5b6413e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:13.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.753+0000 7f5b63fff640 10 monclient: _finish_hunting 0 2024-05-15T12:43:13.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.753+0000 7f5b63fff640 1 monclient: found mon.noname-a 2024-05-15T12:43:13.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.753+0000 7f5b63fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:13.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.753+0000 7f5b63fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.117:3300/0 2024-05-15T12:43:13.744 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.753+0000 7f5b63fff640 1 -- 172.21.15.117:0/1553769346 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b6413f250 con 0x7f5b6413eda0 2024-05-15T12:43:13.746 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.756+0000 7f5b63fff640 1 --2- 172.21.15.117:0/1553769346 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5b6413eda0 0x7f5b641daf10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5b640aa5c0 tx=0x7f5b5c00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:13.746 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.756+0000 7f5b4bfff640 10 client.?.objecter ms_handle_connect 0x7f5b6413eda0 2024-05-15T12:43:13.746 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.756+0000 7f5b4bfff640 10 client.?.objecter resend_mon_ops 2024-05-15T12:43:13.748 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.758+0000 7f3505cb0640 10 monclient: get_monmap_and_config 2024-05-15T12:43:13.748 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.758+0000 7f3505cb0640 10 monclient: build_initial_monmap 2024-05-15T12:43:13.748 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.758+0000 7f3505cb0640 10 monclient: monmap: 2024-05-15T12:43:13.748 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:13.748 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:13.748 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:43:13.759604+0000 2024-05-15T12:43:13.749 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:43:13.759604+0000 2024-05-15T12:43:13.749 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:13.749 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:13.749 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T12:43:13.749 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T12:43:13.749 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T12:43:13.749 INFO:teuthology.orchestra.run.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T12:43:13.749 INFO:teuthology.orchestra.run.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T12:43:13.749 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:13.750 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.759+0000 7f3505cb0640 1 Processor -- start 2024-05-15T12:43:13.750 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.759+0000 7f3505cb0640 1 -- start start 2024-05-15T12:43:13.750 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.759+0000 7f3505cb0640 10 monclient: init 2024-05-15T12:43:13.750 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.760+0000 7f3505cb0640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:13.750 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.760+0000 7f3505cb0640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-05-15T12:43:13.751 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.760+0000 7f3505cb0640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f35000b4d00 0x7f35000b1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:13.751 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.760+0000 7f3505cb0640 10 monclient(hunting): picked mon.noname-c con 0x7f35000b4d00 addr v2:172.21.15.178:3300/0 2024-05-15T12:43:13.751 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.760+0000 7f3505cb0640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f35000b50e0 0x7f3500145c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:13.751 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.760+0000 7f3505cb0640 10 monclient(hunting): picked mon.noname-b con 0x7f35000b50e0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:43:13.751 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.760+0000 7f3505cb0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f35000a94b0 addr v1:172.21.15.178:6789/0 2024-05-15T12:43:13.751 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.760+0000 7f3505cb0640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35000ab1b0 con 0x7f35000a94b0 2024-05-15T12:43:13.751 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.760+0000 7f3505cb0640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:13.751 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.760+0000 7f3505cb0640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35000b23c0 con 0x7f35000b4d00 2024-05-15T12:43:13.751 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.760+0000 7f3505cb0640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:13.751 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.760+0000 7f3505cb0640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35000b16f0 con 0x7f35000b50e0 2024-05-15T12:43:13.751 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.760+0000 7f3505cb0640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:13.751 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.760+0000 7f3505cb0640 10 monclient(hunting): authenticate will time out at 1357.002007s 2024-05-15T12:43:13.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.760+0000 7f5b4bfff640 1 -- 172.21.15.117:0/1553769346 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b5c007030 con 0x7f5b6413eda0 2024-05-15T12:43:13.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.760+0000 7f5b4bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:13.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.760+0000 7f5b4bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T12:43:13.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.760+0000 7f5b4bfff640 10 monclient: dump: 2024-05-15T12:43:13.753 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:13.753 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:13.753 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.753 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.753 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:13.753 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:13.753 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:13.753 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:13.753 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:13.753 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:13.753 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.760+0000 7f5b4bfff640 10 monclient: _finish_auth 0 2024-05-15T12:43:13.753 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.760+0000 7f5b4bfff640 10 monclient: _check_auth_tickets 2024-05-15T12:43:13.753 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.760+0000 7f5b4bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:13.753 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.760+0000 7f5b4bfff640 1 -- 172.21.15.117:0/1553769346 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b5c0071d0 con 0x7f5b6413eda0 2024-05-15T12:43:13.753 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.760+0000 7f5b69f5d640 5 monclient: authenticate success, global_id 4368 2024-05-15T12:43:13.754 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.760+0000 7f5b69f5d640 10 monclient: _renew_subs 2024-05-15T12:43:13.754 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.760+0000 7f5b69f5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:43:13.754 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.760+0000 7f5b69f5d640 1 -- 172.21.15.117:0/1553769346 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b640af650 con 0x7f5b6413eda0 2024-05-15T12:43:13.754 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.760+0000 7f5b4bfff640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:13.754 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.760+0000 7f5b4bfff640 1 -- 172.21.15.117:0/1553769346 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b5c0075c0 con 0x7f5b6413eda0 2024-05-15T12:43:13.754 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.760+0000 7f5b4bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:13.754 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.760+0000 7f5b4bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T12:43:13.754 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.760+0000 7f5b4bfff640 10 monclient: dump: 2024-05-15T12:43:13.754 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:13.754 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:13.754 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.754 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.754 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:13.754 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:13.755 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:13.755 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:13.755 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:13.755 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:13.755 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.760+0000 7f5b61ffb640 4 set_mon_vals no callback set 2024-05-15T12:43:13.755 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.760+0000 7f5b69f5d640 10 client.4368.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:43:13.755 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.760+0000 7f5b69f5d640 10 monclient: _renew_subs 2024-05-15T12:43:13.755 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.760+0000 7f5b69f5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:43:13.755 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.760+0000 7f5b69f5d640 1 -- 172.21.15.117:0/1553769346 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b641db630 con 0x7f5b6413eda0 2024-05-15T12:43:13.755 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.760+0000 7f5b69f5d640 1 librados: init done 2024-05-15T12:43:13.755 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.761+0000 7f5b69f5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:13.755 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.761+0000 7f5b69f5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:13.755 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.761+0000 7f5b69f5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:43:13.755 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.761+0000 7f5b69f5d640 1 -- 172.21.15.117:0/1553769346 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b6404f0e0 con 0x7f5b6413eda0 2024-05-15T12:43:13.779 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.788+0000 7f5b4bfff640 1 -- 172.21.15.117:0/1553769346 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f5b5c007760 con 0x7f5b6413eda0 2024-05-15T12:43:13.779 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.788+0000 7f5b4bfff640 1 --2- 172.21.15.117:0/1553769346 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f5b5006cee0 0x7f5b5006f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:13.779 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.789+0000 7f5b637fe640 1 --2- 172.21.15.117:0/1553769346 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f5b5006cee0 0x7f5b5006f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:13.779 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.789+0000 7f5b637fe640 10 monclient: get_auth_request con 0x7f5b5006cee0 auth_method 0 2024-05-15T12:43:13.779 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.790+0000 7f5b637fe640 1 --2- 172.21.15.117:0/1553769346 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f5b5006cee0 0x7f5b5006f3a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5b540096b0 tx=0x7f5b54006ca0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:13.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.790+0000 7f5b4bfff640 10 client.4368.objecter ms_handle_connect 0x7f5b5006cee0 2024-05-15T12:43:13.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.808+0000 7f35f4ff9640 1 -- 172.21.15.117:0/3546014827 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f35f00570c0 con 0x7f35f813c2a0 2024-05-15T12:43:13.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.808+0000 7f35f4ff9640 10 client.4316.objecter ms_dispatch 0x7f35f8000b90 osd_map(26..26 src has 1..26) v4 2024-05-15T12:43:13.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.808+0000 7f35f4ff9640 3 client.4316.objecter handle_osd_map got epochs [26,26] > 0 2024-05-15T12:43:13.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.808+0000 7f35f4ff9640 3 client.4316.objecter handle_osd_map decoding full epoch 26 2024-05-15T12:43:13.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.808+0000 7f35f4ff9640 20 client.4316.objecter dump_active .. 0 homeless 2024-05-15T12:43:13.848 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.858+0000 7f2b60b87640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2024-05-15T12:43:13.848 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.858+0000 7f2b60b87640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-05-15T12:43:13.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.858+0000 7f2b60b87640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:43:13.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.858+0000 7f2b60b87640 1 -- 172.21.15.117:0/794198670 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f2b5c13eee0 con 0x7f2b5c0b86e0 2024-05-15T12:43:13.873 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.883+0000 7f2eb97fa640 1 -- 172.21.15.117:0/1684657204 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f2eac01a020 con 0x7f2ebc0a94b0 2024-05-15T12:43:13.873 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.883+0000 7f2eb97fa640 1 --2- 172.21.15.117:0/1684657204 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f2e9c06ce90 0x7f2e9c06f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:13.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.883+0000 7f2eb3fff640 1 --2- 172.21.15.117:0/1684657204 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f2e9c06ce90 0x7f2e9c06f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:13.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.883+0000 7f2eb3fff640 10 monclient: get_auth_request con 0x7f2e9c06ce90 auth_method 0 2024-05-15T12:43:13.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.884+0000 7f2eb3fff640 1 --2- 172.21.15.117:0/1684657204 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f2e9c06ce90 0x7f2e9c06f350 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc13d5a0 tx=0x7f2ea4006c90 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:13.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.884+0000 7f2eb97fa640 10 client.4375.objecter ms_handle_connect 0x7f2e9c06ce90 2024-05-15T12:43:13.876 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.886+0000 7f2eb97fa640 1 -- 172.21.15.117:0/1684657204 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f2eac056830 con 0x7f2ebc0a94b0 2024-05-15T12:43:13.876 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.886+0000 7f2eb97fa640 10 client.4375.objecter ms_dispatch 0x7f2ebc000b90 osd_map(26..26 src has 1..26) v4 2024-05-15T12:43:13.876 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.886+0000 7f2eb97fa640 3 client.4375.objecter handle_osd_map got epochs [26,26] > 0 2024-05-15T12:43:13.876 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.886+0000 7f2eb97fa640 3 client.4375.objecter handle_osd_map decoding full epoch 26 2024-05-15T12:43:13.876 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.886+0000 7f2eb97fa640 20 client.4375.objecter dump_active .. 0 homeless 2024-05-15T12:43:13.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.898+0000 7f2eb97fa640 1 -- 172.21.15.117:0/1684657204 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f2eac021070 con 0x7f2ebc0a94b0 2024-05-15T12:43:13.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.898+0000 7f2eb97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:13.889 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.898+0000 7f2eb97fa640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:43:13.893 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.903+0000 7f139d304640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-05-15T12:43:13.893 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.903+0000 7f139d304640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-05-15T12:43:13.893 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.903+0000 7f139d304640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:43:13.893 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.903+0000 7f139d304640 1 -- 172.21.15.117:0/4276070227 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f13980b3730 con 0x7f13980ae830 2024-05-15T12:43:13.906 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.916+0000 7f34ff7fe640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f35000b4d00 0x7f35000b1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:13.912 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.922+0000 7f34ff7fe640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f35000b4d00 0x7f35000b1e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:42666/0 (socket says 172.21.15.117:42666) 2024-05-15T12:43:13.912 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.922+0000 7f34ff7fe640 1 -- 172.21.15.117:0/3645613686 learned_addr learned my addr 172.21.15.117:0/3645613686 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:43:13.912 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.922+0000 7f34ff7fe640 10 monclient(hunting): get_auth_request con 0x7f35000b4d00 auth_method 0 2024-05-15T12:43:13.912 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.922+0000 7f34ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:13.912 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.922+0000 7f34ff7fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:13.912 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.922+0000 7f34ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:13.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.923+0000 7f34ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:13.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.923+0000 7f34ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:13.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.923+0000 7f34ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:13.914 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.924+0000 7f34ff7fe640 10 monclient(hunting): handle_auth_done global_id 4378 payload 274 2024-05-15T12:43:13.914 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.924+0000 7f34ff7fe640 1 -- 172.21.15.117:0/3645613686 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f35000b50e0 msgr2=0x7f3500145c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:43:13.915 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.924+0000 7f34ff7fe640 1 --2- 172.21.15.117:0/3645613686 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f35000b50e0 0x7f3500145c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:13.915 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.924+0000 7f34ff7fe640 1 -- 172.21.15.117:0/3645613686 >> v1:172.21.15.178:6789/0 conn(0x7f35000a94b0 legacy=0x7f35000b0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:43:13.915 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.924+0000 7f34ff7fe640 10 monclient: _finish_hunting 0 2024-05-15T12:43:13.915 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.924+0000 7f34ff7fe640 1 monclient: found mon.noname-c 2024-05-15T12:43:13.915 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.924+0000 7f34ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:13.915 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.924+0000 7f34ff7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.178:3300/0 2024-05-15T12:43:13.915 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.924+0000 7f34ff7fe640 1 -- 172.21.15.117:0/3645613686 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3500146160 con 0x7f35000b4d00 2024-05-15T12:43:13.918 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.928+0000 7f34ff7fe640 1 --2- 172.21.15.117:0/3645613686 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f35000b4d00 0x7f35000b1e80 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f34f00098a0 tx=0x7f34f002b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae619122557b5b89 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:13.922 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.932+0000 7f2b4affd640 1 -- 172.21.15.117:0/794198670 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2b5005d220 con 0x7f2b5c0b86e0 2024-05-15T12:43:13.922 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.932+0000 7f2b4affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-05-15T12:43:13.922 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.932+0000 7f2b4affd640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:43:13.922 INFO:teuthology.orchestra.run.smithi117.stdout:55834574855 2024-05-15T12:43:13.924 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.934+0000 7f2b60b87640 10 librados: watch_flush enter 2024-05-15T12:43:13.925 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.934+0000 7f2b60b87640 10 librados: watch_flush exit 2024-05-15T12:43:13.925 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.934+0000 7f2b60b87640 20 client.4366.objecter shutdown clearing up homeless session... 2024-05-15T12:43:13.925 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.934+0000 7f2b60b87640 10 client.4366.objecter successfully canceled tick 2024-05-15T12:43:13.925 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.934+0000 7f2b60b87640 1 -- 172.21.15.117:0/794198670 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f2b3c06ceb0 msgr2=0x7f2b3c06f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:13.925 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.934+0000 7f2b60b87640 1 --2- 172.21.15.117:0/794198670 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f2b3c06ceb0 0x7f2b3c06f370 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c036000 tx=0x7f2b4c030000 comp rx=0 tx=0).stop 2024-05-15T12:43:13.925 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.934+0000 7f2b60b87640 10 monclient: shutdown 2024-05-15T12:43:13.925 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.934+0000 7f2b60b87640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:13.925 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.934+0000 7f2b60b87640 1 -- 172.21.15.117:0/794198670 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2b5c0b86e0 msgr2=0x7f2b5c1429f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:13.925 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.934+0000 7f2b60b87640 1 --2- 172.21.15.117:0/794198670 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2b5c0b86e0 0x7f2b5c1429f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2b5000e9b0 tx=0x7f2b5000ee70 comp rx=0 tx=0).stop 2024-05-15T12:43:13.925 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.934+0000 7f2b60b87640 1 -- 172.21.15.117:0/794198670 shutdown_connections 2024-05-15T12:43:13.925 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.934+0000 7f2b60b87640 1 --2- 172.21.15.117:0/794198670 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b5c13f870 0x7f2b5c142f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:13.925 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.934+0000 7f2b60b87640 1 --2- 172.21.15.117:0/794198670 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f2b3c06ceb0 0x7f2b3c06f370 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:13.925 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.934+0000 7f2b60b87640 1 --2- 172.21.15.117:0/794198670 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2b5c0b86e0 0x7f2b5c1429f0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:13.926 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.934+0000 7f2b60b87640 1 --2- 172.21.15.117:0/794198670 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2b5c0b1dd0 0x7f2b5c145360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:13.926 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.934+0000 7f2b60b87640 1 -- 172.21.15.117:0/794198670 >> 172.21.15.117:0/794198670 conn(0x7f2b5c066f00 msgr2=0x7f2b5c0672e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:13.926 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.934+0000 7f2b60b87640 1 -- 172.21.15.117:0/794198670 shutdown_connections 2024-05-15T12:43:13.926 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.934+0000 7f34fe7fc640 1 -- 172.21.15.117:0/3645613686 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f00317c0 con 0x7f35000b4d00 2024-05-15T12:43:13.926 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.934+0000 7f2b60b87640 1 -- 172.21.15.117:0/794198670 wait complete. 2024-05-15T12:43:13.927 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.934+0000 7f34fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:13.927 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.934+0000 7f34fe7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:43:13.927 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.934+0000 7f34fe7fc640 10 monclient: dump: 2024-05-15T12:43:13.927 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:13.927 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:13.928 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.928 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.928 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:13.928 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:13.928 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:13.928 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:13.928 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:13.928 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:13.928 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.934+0000 7f34fe7fc640 10 monclient: _finish_auth 0 2024-05-15T12:43:13.928 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.934+0000 7f34fe7fc640 10 monclient: _check_auth_tickets 2024-05-15T12:43:13.928 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.934+0000 7f34fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:13.928 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.934+0000 7f34fe7fc640 1 -- 172.21.15.117:0/3645613686 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34f0037070 con 0x7f35000b4d00 2024-05-15T12:43:13.928 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.934+0000 7f34fe7fc640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:13.929 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.934+0000 7f34fe7fc640 1 -- 172.21.15.117:0/3645613686 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f0031b80 con 0x7f35000b4d00 2024-05-15T12:43:13.929 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.934+0000 7f34fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:13.929 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.934+0000 7f34fe7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:43:13.929 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.934+0000 7f34fe7fc640 10 monclient: dump: 2024-05-15T12:43:13.929 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:13.929 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:13.929 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.929 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.929 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:13.929 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:13.929 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:13.929 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:13.929 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:13.929 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:13.929 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.934+0000 7f3505cb0640 5 monclient: authenticate success, global_id 4378 2024-05-15T12:43:13.930 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.934+0000 7f3505cb0640 10 monclient: get_monmap_and_config success 2024-05-15T12:43:13.930 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.934+0000 7f3505cb0640 4 set_mon_vals no callback set 2024-05-15T12:43:13.930 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.934+0000 7f2b60b87640 1 librados: shutdown 2024-05-15T12:43:13.932 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.935+0000 7f3505cb0640 10 monclient: shutdown 2024-05-15T12:43:13.933 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.935+0000 7f3505cb0640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:13.933 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.935+0000 7f3505cb0640 1 -- 172.21.15.117:0/3645613686 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f35000b4d00 msgr2=0x7f35000b1e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:13.933 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.935+0000 7f3505cb0640 1 --2- 172.21.15.117:0/3645613686 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f35000b4d00 0x7f35000b1e80 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f34f00098a0 tx=0x7f34f002b860 comp rx=0 tx=0).stop 2024-05-15T12:43:13.933 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.935+0000 7f3505cb0640 1 -- 172.21.15.117:0/3645613686 shutdown_connections 2024-05-15T12:43:13.933 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.935+0000 7f3505cb0640 1 --2- 172.21.15.117:0/3645613686 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f35000b50e0 0x7f3500145c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:13.933 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.935+0000 7f3505cb0640 1 --2- 172.21.15.117:0/3645613686 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f35000b4d00 0x7f35000b1e80 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:13.933 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.935+0000 7f3505cb0640 1 -- 172.21.15.117:0/3645613686 >> 172.21.15.117:0/3645613686 conn(0x7f3500066cc0 msgr2=0x7f35000670d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:13.933 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.935+0000 7f3505cb0640 1 -- 172.21.15.117:0/3645613686 shutdown_connections 2024-05-15T12:43:13.933 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.935+0000 7f3505cb0640 1 -- 172.21.15.117:0/3645613686 wait complete. 2024-05-15T12:43:13.933 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.936+0000 7f3505cb0640 10 monclient: build_initial_monmap 2024-05-15T12:43:13.933 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.936+0000 7f3505cb0640 10 monclient: monmap: 2024-05-15T12:43:13.933 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:13.934 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:43:13.934 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-05-15T12:43:13.934 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-05-15T12:43:13.934 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:13.934 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:13.934 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T12:43:13.934 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T12:43:13.934 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T12:43:13.934 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:13.934 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.936+0000 7f3505cb0640 1 librados: starting msgr at 2024-05-15T12:43:13.934 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.936+0000 7f3505cb0640 1 librados: starting objecter 2024-05-15T12:43:13.934 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.936+0000 7f3505cb0640 1 Processor -- start 2024-05-15T12:43:13.934 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.936+0000 7f3505cb0640 1 -- start start 2024-05-15T12:43:13.934 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.936+0000 7f3505cb0640 1 librados: setting wanted keys 2024-05-15T12:43:13.935 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.936+0000 7f3505cb0640 1 librados: calling monclient init 2024-05-15T12:43:13.935 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.936+0000 7f3505cb0640 10 monclient: init 2024-05-15T12:43:13.935 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.937+0000 7f3505cb0640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:13.935 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.937+0000 7f3505cb0640 10 monclient: _add_conns ranks=[0,2,1] 2024-05-15T12:43:13.935 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.937+0000 7f3505cb0640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f35000a94b0 0x7f3500142ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:13.935 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.937+0000 7f3505cb0640 10 monclient(hunting): picked mon.noname-a con 0x7f35000a94b0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T12:43:13.935 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.937+0000 7f3505cb0640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f35000b4d00 0x7f350013e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:13.935 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.937+0000 7f3505cb0640 10 monclient(hunting): picked mon.noname-c con 0x7f35000b4d00 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:43:13.935 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.937+0000 7f3505cb0640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f35000affa0 0x7f35001dbef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:13.935 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.937+0000 7f3505cb0640 10 monclient(hunting): picked mon.noname-b con 0x7f35000affa0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:43:13.935 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.937+0000 7f3505cb0640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:13.935 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.937+0000 7f3505cb0640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f350014cfb0 con 0x7f35000a94b0 2024-05-15T12:43:13.935 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.937+0000 7f3505cb0640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:13.935 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.937+0000 7f3505cb0640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f350014d130 con 0x7f35000affa0 2024-05-15T12:43:13.936 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.937+0000 7f3505cb0640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:13.936 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.937+0000 7f3505cb0640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f350014d2b0 con 0x7f35000b4d00 2024-05-15T12:43:13.936 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.937+0000 7f3505cb0640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:13.936 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.937+0000 7f3505cb0640 10 monclient(hunting): authenticate will time out at 1357.179376s 2024-05-15T12:43:13.936 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.938+0000 7f34fffff640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f35000affa0 0x7f35001dbef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:13.936 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.939+0000 7f34fffff640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f35000affa0 0x7f35001dbef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:42682/0 (socket says 172.21.15.117:42682) 2024-05-15T12:43:13.936 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.939+0000 7f34fffff640 1 -- 172.21.15.117:0/3739604034 learned_addr learned my addr 172.21.15.117:0/3739604034 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:43:13.936 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.939+0000 7f34fffff640 10 monclient(hunting): get_auth_request con 0x7f35000affa0 auth_method 0 2024-05-15T12:43:13.936 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.939+0000 7f34fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:13.936 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.939+0000 7f34fffff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:13.936 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.939+0000 7f34fffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:13.936 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.939+0000 7f34fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:13.936 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.939+0000 7f34fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:13.936 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.939+0000 7f34fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:13.936 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.942+0000 7f34fffff640 10 monclient(hunting): handle_auth_done global_id 4381 payload 824 2024-05-15T12:43:13.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.942+0000 7f34fffff640 1 -- 172.21.15.117:0/3739604034 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f35000b4d00 msgr2=0x7f350013e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:13.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.942+0000 7f34fffff640 1 --2- 172.21.15.117:0/3739604034 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f35000b4d00 0x7f350013e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:13.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.942+0000 7f34fffff640 1 -- 172.21.15.117:0/3739604034 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f35000a94b0 msgr2=0x7f3500142ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:13.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.942+0000 7f34fffff640 1 --2- 172.21.15.117:0/3739604034 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f35000a94b0 0x7f3500142ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:13.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.942+0000 7f34fffff640 10 monclient: _finish_hunting 0 2024-05-15T12:43:13.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.942+0000 7f34fffff640 1 monclient: found mon.noname-b 2024-05-15T12:43:13.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.942+0000 7f34fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:13.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.942+0000 7f34fffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2024-05-15T12:43:13.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.942+0000 7f34fffff640 1 -- 172.21.15.117:0/3739604034 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35000ac4d0 con 0x7f35000affa0 2024-05-15T12:43:13.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.945+0000 7f34fffff640 1 --2- 172.21.15.117:0/3739604034 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f35000affa0 0x7f35001dbef0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f34f800b9e0 tx=0x7f34f800bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:13.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.945+0000 7f34e7fff640 10 client.?.objecter ms_handle_connect 0x7f35000affa0 2024-05-15T12:43:13.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.945+0000 7f34e7fff640 10 client.?.objecter resend_mon_ops 2024-05-15T12:43:13.939 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.7 2024-05-15T12:43:13.939 DEBUG:teuthology.parallel:result is None 2024-05-15T12:43:13.940 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.948+0000 7f34e7fff640 1 -- 172.21.15.117:0/3739604034 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f8019070 con 0x7f35000affa0 2024-05-15T12:43:13.940 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.948+0000 7f34e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:13.941 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.948+0000 7f34e7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:43:13.941 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.948+0000 7f34e7fff640 10 monclient: dump: 2024-05-15T12:43:13.941 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:13.941 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:13.941 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.941 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.941 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:13.941 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:13.941 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:13.941 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:13.941 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:13.941 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:13.941 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.948+0000 7f34e7fff640 10 monclient: _finish_auth 0 2024-05-15T12:43:13.941 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.948+0000 7f34e7fff640 10 monclient: _check_auth_tickets 2024-05-15T12:43:13.942 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.948+0000 7f34e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:13.942 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.948+0000 7f34e7fff640 1 -- 172.21.15.117:0/3739604034 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34f800caa0 con 0x7f35000affa0 2024-05-15T12:43:13.942 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.948+0000 7f3505cb0640 5 monclient: authenticate success, global_id 4381 2024-05-15T12:43:13.942 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.948+0000 7f3505cb0640 10 monclient: _renew_subs 2024-05-15T12:43:13.942 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.948+0000 7f3505cb0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:43:13.942 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.948+0000 7f3505cb0640 1 -- 172.21.15.117:0/3739604034 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35000ac7b0 con 0x7f35000affa0 2024-05-15T12:43:13.942 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.948+0000 7f34e7fff640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:13.942 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.948+0000 7f34e7fff640 1 -- 172.21.15.117:0/3739604034 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f8014440 con 0x7f35000affa0 2024-05-15T12:43:13.942 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.948+0000 7f34e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:13.942 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.948+0000 7f34e7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:43:13.942 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.948+0000 7f34e7fff640 10 monclient: dump: 2024-05-15T12:43:13.942 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:13.942 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:13.942 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.942 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:13.943 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:13.943 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:13.943 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:13.943 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:13.943 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:13.943 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:13.943 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.948+0000 7f3505cb0640 10 client.4381.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:43:13.943 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.948+0000 7f3505cb0640 10 monclient: _renew_subs 2024-05-15T12:43:13.943 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.948+0000 7f3505cb0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:43:13.943 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.948+0000 7f34fdffb640 4 set_mon_vals no callback set 2024-05-15T12:43:13.943 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.948+0000 7f3505cb0640 1 -- 172.21.15.117:0/3739604034 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35001dc430 con 0x7f35000affa0 2024-05-15T12:43:13.943 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.948+0000 7f3505cb0640 1 librados: init done 2024-05-15T12:43:13.943 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.949+0000 7f3505cb0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:13.943 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.949+0000 7f3505cb0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:13.943 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.949+0000 7f3505cb0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:43:13.943 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:13.949+0000 7f3505cb0640 1 -- 172.21.15.117:0/3739604034 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f350004f0e0 con 0x7f35000affa0 2024-05-15T12:43:14.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.034+0000 7f35f4ff9640 1 -- 172.21.15.117:0/3546014827 <== mon.2 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f35f80b5b40 con 0x7f35f813c2a0 2024-05-15T12:43:14.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.034+0000 7f35f4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:14.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.034+0000 7f35f4ff9640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:43:14.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.040+0000 7f5b4bfff640 1 -- 172.21.15.117:0/1553769346 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f5b5c0562f0 con 0x7f5b6413eda0 2024-05-15T12:43:14.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.040+0000 7f5b4bfff640 10 client.4368.objecter ms_dispatch 0x7f5b64000b90 osd_map(26..26 src has 1..26) v4 2024-05-15T12:43:14.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.040+0000 7f5b4bfff640 3 client.4368.objecter handle_osd_map got epochs [26,26] > 0 2024-05-15T12:43:14.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.040+0000 7f5b4bfff640 3 client.4368.objecter handle_osd_map decoding full epoch 26 2024-05-15T12:43:14.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.040+0000 7f5b4bfff640 20 client.4368.objecter dump_active .. 0 homeless 2024-05-15T12:43:14.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.040+0000 7f5b4bfff640 1 -- 172.21.15.117:0/1553769346 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f5b5c058950 con 0x7f5b6413eda0 2024-05-15T12:43:14.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.040+0000 7f5b4bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:14.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.040+0000 7f5b4bfff640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:43:14.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.040+0000 7f8a6affd640 1 -- 172.21.15.117:0/3102839906 <== mon.2 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8a700582e0 con 0x7f8a7406c900 2024-05-15T12:43:14.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.040+0000 7f8a6affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-05-15T12:43:14.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.040+0000 7f8a6affd640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:43:14.031 INFO:teuthology.orchestra.run.smithi117.stdout:85899345931 2024-05-15T12:43:14.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.042+0000 7f8a7c5aa640 10 librados: watch_flush enter 2024-05-15T12:43:14.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.042+0000 7f8a7c5aa640 10 librados: watch_flush exit 2024-05-15T12:43:14.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.042+0000 7f8a7c5aa640 20 client.4310.objecter shutdown clearing up homeless session... 2024-05-15T12:43:14.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.042+0000 7f8a7c5aa640 10 client.4310.objecter successfully canceled tick 2024-05-15T12:43:14.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.042+0000 7f8a7c5aa640 1 -- 172.21.15.117:0/3102839906 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f8a5806ce60 msgr2=0x7f8a5806f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:14.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.042+0000 7f8a7c5aa640 1 --2- 172.21.15.117:0/3102839906 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f8a5806ce60 0x7f8a5806f320 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8a64036000 tx=0x7f8a6403d040 comp rx=0 tx=0).stop 2024-05-15T12:43:14.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.043+0000 7f8a7c5aa640 10 monclient: shutdown 2024-05-15T12:43:14.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.043+0000 7f8a7c5aa640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:14.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.043+0000 7f8a7c5aa640 1 -- 172.21.15.117:0/3102839906 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8a7406c900 msgr2=0x7f8a740b5be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:14.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.043+0000 7f8a7c5aa640 1 --2- 172.21.15.117:0/3102839906 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8a7406c900 0x7f8a740b5be0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8a7000c370 tx=0x7f8a7000c830 comp rx=0 tx=0).stop 2024-05-15T12:43:14.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.043+0000 7f8a7c5aa640 1 -- 172.21.15.117:0/3102839906 shutdown_connections 2024-05-15T12:43:14.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.043+0000 7f8a7c5aa640 1 --2- 172.21.15.117:0/3102839906 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8a741431b0 0x7f8a7413de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:14.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.043+0000 7f8a7c5aa640 1 --2- 172.21.15.117:0/3102839906 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f8a5806ce60 0x7f8a5806f320 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:14.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.043+0000 7f8a7c5aa640 1 --2- 172.21.15.117:0/3102839906 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8a740af1c0 0x7f8a741d8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:14.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.043+0000 7f8a7c5aa640 1 --2- 172.21.15.117:0/3102839906 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8a7406c900 0x7f8a740b5be0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:14.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.043+0000 7f8a7c5aa640 1 -- 172.21.15.117:0/3102839906 >> 172.21.15.117:0/3102839906 conn(0x7f8a740b2ac0 msgr2=0x7f8a740b0f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:14.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.043+0000 7f8a7c5aa640 1 -- 172.21.15.117:0/3102839906 shutdown_connections 2024-05-15T12:43:14.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.043+0000 7f8a7c5aa640 1 -- 172.21.15.117:0/3102839906 wait complete. 2024-05-15T12:43:14.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.043+0000 7f8a7c5aa640 1 librados: shutdown 2024-05-15T12:43:14.039 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.049+0000 7f138f7fe640 1 -- 172.21.15.117:0/4276070227 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f13900591b0 con 0x7f13980ae830 2024-05-15T12:43:14.039 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.049+0000 7f138f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-05-15T12:43:14.039 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.049+0000 7f138f7fe640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:43:14.039 INFO:teuthology.orchestra.run.smithi117.stdout:85899345929 2024-05-15T12:43:14.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.052+0000 7f139d304640 10 librados: watch_flush enter 2024-05-15T12:43:14.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.052+0000 7f139d304640 10 librados: watch_flush exit 2024-05-15T12:43:14.044 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.052+0000 7f139d304640 20 client.4362.objecter shutdown clearing up homeless session... 2024-05-15T12:43:14.044 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.052+0000 7f139d304640 10 client.4362.objecter successfully canceled tick 2024-05-15T12:43:14.044 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.052+0000 7f139d304640 1 -- 172.21.15.117:0/4276070227 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f137806ce60 msgr2=0x7f137806f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:14.044 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.052+0000 7f139d304640 1 --2- 172.21.15.117:0/4276070227 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f137806ce60 0x7f137806f320 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1388036000 tx=0x7f1388030000 comp rx=0 tx=0).stop 2024-05-15T12:43:14.044 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.052+0000 7f139d304640 10 monclient: shutdown 2024-05-15T12:43:14.044 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.052+0000 7f139d304640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:14.044 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.052+0000 7f139d304640 1 -- 172.21.15.117:0/4276070227 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13980ae830 msgr2=0x7f13981396d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:14.044 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.052+0000 7f139d304640 1 --2- 172.21.15.117:0/4276070227 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13980ae830 0x7f13981396d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f139000b9e0 tx=0x7f139000bea0 comp rx=0 tx=0).stop 2024-05-15T12:43:14.044 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.052+0000 7f139d304640 1 -- 172.21.15.117:0/4276070227 shutdown_connections 2024-05-15T12:43:14.044 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.052+0000 7f139d304640 1 --2- 172.21.15.117:0/4276070227 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13980ae830 0x7f13981396d0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:14.044 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.052+0000 7f139d304640 1 --2- 172.21.15.117:0/4276070227 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f137806ce60 0x7f137806f320 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:14.044 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.052+0000 7f139d304640 1 --2- 172.21.15.117:0/4276070227 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f13980b4d00 0x7f1398139190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:14.044 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.052+0000 7f139d304640 1 --2- 172.21.15.117:0/4276070227 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f13980a94b0 0x7f13980abdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:14.045 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.052+0000 7f139d304640 1 -- 172.21.15.117:0/4276070227 >> 172.21.15.117:0/4276070227 conn(0x7f1398066cc0 msgr2=0x7f139813b660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:14.045 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.052+0000 7f139d304640 1 -- 172.21.15.117:0/4276070227 shutdown_connections 2024-05-15T12:43:14.045 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.053+0000 7f139d304640 1 -- 172.21.15.117:0/4276070227 wait complete. 2024-05-15T12:43:14.045 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.053+0000 7f139d304640 1 librados: shutdown 2024-05-15T12:43:14.048 INFO:tasks.ceph.ceph_manager.ceph:need seq 85899345931 got 85899345931 for osd.0 2024-05-15T12:43:14.048 DEBUG:teuthology.parallel:result is None 2024-05-15T12:43:14.059 INFO:tasks.ceph.ceph_manager.ceph:need seq 85899345929 got 85899345929 for osd.1 2024-05-15T12:43:14.059 DEBUG:teuthology.parallel:result is None 2024-05-15T12:43:14.187 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.197+0000 7f34e7fff640 1 -- 172.21.15.117:0/3739604034 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f34f801c070 con 0x7f35000affa0 2024-05-15T12:43:14.187 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.197+0000 7f34e7fff640 1 --2- 172.21.15.117:0/3739604034 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f34ec06ceb0 0x7f34ec06f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:14.187 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.197+0000 7f34ff7fe640 1 --2- 172.21.15.117:0/3739604034 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f34ec06ceb0 0x7f34ec06f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:14.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.197+0000 7f34ff7fe640 10 monclient: get_auth_request con 0x7f34ec06ceb0 auth_method 0 2024-05-15T12:43:14.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.198+0000 7f34e7fff640 10 client.4381.objecter ms_handle_connect 0x7f34ec06ceb0 2024-05-15T12:43:14.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.198+0000 7f34ff7fe640 1 --2- 172.21.15.117:0/3739604034 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f34ec06ceb0 0x7f34ec06f370 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f34f0036000 tx=0x7f34f0030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:14.190 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.200+0000 7f2ec29a3640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-05-15T12:43:14.190 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.200+0000 7f2ec29a3640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-05-15T12:43:14.190 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.200+0000 7f2ec29a3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:43:14.190 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.200+0000 7f2ec29a3640 1 -- 172.21.15.117:0/1684657204 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f2ebc0b7540 con 0x7f2ebc0a94b0 2024-05-15T12:43:14.191 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.200+0000 7f34e7fff640 1 -- 172.21.15.117:0/3739604034 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f34f8056b20 con 0x7f35000affa0 2024-05-15T12:43:14.191 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.200+0000 7f34e7fff640 10 client.4381.objecter ms_dispatch 0x7f3500000b90 osd_map(26..26 src has 1..26) v4 2024-05-15T12:43:14.191 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.200+0000 7f34e7fff640 3 client.4381.objecter handle_osd_map got epochs [26,26] > 0 2024-05-15T12:43:14.191 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.200+0000 7f34e7fff640 3 client.4381.objecter handle_osd_map decoding full epoch 26 2024-05-15T12:43:14.191 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.200+0000 7f34e7fff640 20 client.4381.objecter dump_active .. 0 homeless 2024-05-15T12:43:14.202 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.212+0000 7f34e7fff640 1 -- 172.21.15.117:0/3739604034 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f34f8023020 con 0x7f35000affa0 2024-05-15T12:43:14.202 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.212+0000 7f34e7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:14.202 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.212+0000 7f34e7fff640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:43:14.268 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.277+0000 7f5b69f5d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-05-15T12:43:14.268 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.277+0000 7f5b69f5d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-05-15T12:43:14.268 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.278+0000 7f5b69f5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:43:14.268 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.278+0000 7f5b69f5d640 1 -- 172.21.15.117:0/1553769346 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f5b640b3b20 con 0x7f5b6413eda0 2024-05-15T12:43:14.272 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.282+0000 7f5b4bfff640 1 -- 172.21.15.117:0/1553769346 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5b5c058250 con 0x7f5b6413eda0 2024-05-15T12:43:14.272 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.282+0000 7f5b4bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-05-15T12:43:14.272 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.282+0000 7f5b4bfff640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:43:14.272 INFO:teuthology.orchestra.run.smithi117.stdout:60129542151 2024-05-15T12:43:14.275 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.284+0000 7f5b69f5d640 10 librados: watch_flush enter 2024-05-15T12:43:14.275 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.284+0000 7f5b69f5d640 10 librados: watch_flush exit 2024-05-15T12:43:14.275 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.284+0000 7f5b69f5d640 20 client.4368.objecter shutdown clearing up homeless session... 2024-05-15T12:43:14.275 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.284+0000 7f5b69f5d640 10 client.4368.objecter successfully canceled tick 2024-05-15T12:43:14.275 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.284+0000 7f5b69f5d640 1 -- 172.21.15.117:0/1553769346 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f5b5006cee0 msgr2=0x7f5b5006f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:14.275 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.284+0000 7f5b69f5d640 1 --2- 172.21.15.117:0/1553769346 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f5b5006cee0 0x7f5b5006f3a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5b540096b0 tx=0x7f5b54006ca0 comp rx=0 tx=0).stop 2024-05-15T12:43:14.275 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.284+0000 7f5b69f5d640 10 monclient: shutdown 2024-05-15T12:43:14.275 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.284+0000 7f5b69f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:14.275 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.284+0000 7f5b69f5d640 1 -- 172.21.15.117:0/1553769346 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5b6413eda0 msgr2=0x7f5b641daf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:14.276 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.284+0000 7f5b69f5d640 1 --2- 172.21.15.117:0/1553769346 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5b6413eda0 0x7f5b641daf10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5b640aa5c0 tx=0x7f5b5c00c960 comp rx=0 tx=0).stop 2024-05-15T12:43:14.276 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.284+0000 7f5b69f5d640 1 -- 172.21.15.117:0/1553769346 shutdown_connections 2024-05-15T12:43:14.276 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.284+0000 7f5b69f5d640 1 --2- 172.21.15.117:0/1553769346 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5b6413eda0 0x7f5b641daf10 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:14.276 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.284+0000 7f5b69f5d640 1 --2- 172.21.15.117:0/1553769346 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f5b5006cee0 0x7f5b5006f3a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:14.276 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.284+0000 7f5b69f5d640 1 --2- 172.21.15.117:0/1553769346 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5b6413fa30 0x7f5b6413e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:14.276 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.284+0000 7f5b69f5d640 1 --2- 172.21.15.117:0/1553769346 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5b640aeb80 0x7f5b640af110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:14.276 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.284+0000 7f5b69f5d640 1 -- 172.21.15.117:0/1553769346 >> 172.21.15.117:0/1553769346 conn(0x7f5b64066cc0 msgr2=0x7f5b64145070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:14.276 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.284+0000 7f5b69f5d640 1 -- 172.21.15.117:0/1553769346 shutdown_connections 2024-05-15T12:43:14.276 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.285+0000 7f5b69f5d640 1 -- 172.21.15.117:0/1553769346 wait complete. 2024-05-15T12:43:14.276 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.285+0000 7f5b69f5d640 1 librados: shutdown 2024-05-15T12:43:14.278 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.287+0000 7f2eb97fa640 1 -- 172.21.15.117:0/1684657204 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2ebc0b7540 con 0x7f2ebc0a94b0 2024-05-15T12:43:14.278 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.287+0000 7f2eb97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-05-15T12:43:14.278 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.287+0000 7f2eb97fa640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:43:14.278 INFO:teuthology.orchestra.run.smithi117.stdout:85899345931 2024-05-15T12:43:14.281 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.291+0000 7f2ec29a3640 10 librados: watch_flush enter 2024-05-15T12:43:14.281 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.291+0000 7f2ec29a3640 10 librados: watch_flush exit 2024-05-15T12:43:14.282 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.291+0000 7f2ec29a3640 20 client.4375.objecter shutdown clearing up homeless session... 2024-05-15T12:43:14.282 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.291+0000 7f2ec29a3640 10 client.4375.objecter successfully canceled tick 2024-05-15T12:43:14.282 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.291+0000 7f2ec29a3640 1 -- 172.21.15.117:0/1684657204 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f2e9c06ce90 msgr2=0x7f2e9c06f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:14.282 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.291+0000 7f2ec29a3640 1 --2- 172.21.15.117:0/1684657204 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f2e9c06ce90 0x7f2e9c06f350 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc13d5a0 tx=0x7f2ea4006c90 comp rx=0 tx=0).stop 2024-05-15T12:43:14.282 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.291+0000 7f2ec29a3640 10 monclient: shutdown 2024-05-15T12:43:14.282 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.291+0000 7f2ec29a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:14.282 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.291+0000 7f2ec29a3640 1 -- 172.21.15.117:0/1684657204 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2ebc0a94b0 msgr2=0x7f2ebc1422c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:14.282 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.291+0000 7f2ec29a3640 1 --2- 172.21.15.117:0/1684657204 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2ebc0a94b0 0x7f2ebc1422c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2eac000f60 tx=0x7f2eac00c800 comp rx=0 tx=0).stop 2024-05-15T12:43:14.282 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.291+0000 7f2ec29a3640 1 -- 172.21.15.117:0/1684657204 shutdown_connections 2024-05-15T12:43:14.282 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.291+0000 7f2ec29a3640 1 --2- 172.21.15.117:0/1684657204 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2ebc13c2a0 0x7f2ebc13e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:14.282 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.291+0000 7f2ec29a3640 1 --2- 172.21.15.117:0/1684657204 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2ebc0b4d00 0x7f2ebc13d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:14.282 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.291+0000 7f2ec29a3640 1 --2- 172.21.15.117:0/1684657204 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f2e9c06ce90 0x7f2e9c06f350 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:14.282 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.291+0000 7f2ec29a3640 1 --2- 172.21.15.117:0/1684657204 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2ebc0a94b0 0x7f2ebc1422c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:14.282 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.291+0000 7f2ec29a3640 1 -- 172.21.15.117:0/1684657204 >> 172.21.15.117:0/1684657204 conn(0x7f2ebc066cc0 msgr2=0x7f2ebc0b86c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:14.283 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.291+0000 7f2ec29a3640 1 -- 172.21.15.117:0/1684657204 shutdown_connections 2024-05-15T12:43:14.283 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.291+0000 7f2ec29a3640 1 -- 172.21.15.117:0/1684657204 wait complete. 2024-05-15T12:43:14.283 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.291+0000 7f2ec29a3640 1 librados: shutdown 2024-05-15T12:43:14.286 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542151 for osd.5 2024-05-15T12:43:14.286 DEBUG:teuthology.parallel:result is None 2024-05-15T12:43:14.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.303+0000 7f35fd934640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-05-15T12:43:14.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.303+0000 7f35fd934640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-05-15T12:43:14.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.303+0000 7f35fd934640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T12:43:14.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.303+0000 7f35fd934640 1 -- 172.21.15.117:0/3546014827 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f35f804f0e0 con 0x7f35f813c2a0 2024-05-15T12:43:14.298 INFO:tasks.ceph.ceph_manager.ceph:need seq 85899345930 got 85899345931 for osd.2 2024-05-15T12:43:14.299 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.308+0000 7f35f4ff9640 1 -- 172.21.15.117:0/3546014827 <== mon.2 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f35f0059720 con 0x7f35f813c2a0 2024-05-15T12:43:14.299 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.308+0000 7f35f4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-05-15T12:43:14.299 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.308+0000 7f35f4ff9640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:43:14.299 INFO:teuthology.orchestra.run.smithi117.stdout:81604378634 2024-05-15T12:43:14.299 DEBUG:teuthology.parallel:result is None 2024-05-15T12:43:14.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.311+0000 7f35fd934640 10 librados: watch_flush enter 2024-05-15T12:43:14.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.311+0000 7f35fd934640 10 librados: watch_flush exit 2024-05-15T12:43:14.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.311+0000 7f35fd934640 20 client.4316.objecter shutdown clearing up homeless session... 2024-05-15T12:43:14.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.311+0000 7f35fd934640 10 client.4316.objecter successfully canceled tick 2024-05-15T12:43:14.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.311+0000 7f35fd934640 1 -- 172.21.15.117:0/3546014827 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f35d806ce60 msgr2=0x7f35d806f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:14.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.311+0000 7f35fd934640 1 --2- 172.21.15.117:0/3546014827 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f35d806ce60 0x7f35d806f320 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f35e8009710 tx=0x7f35e8006c90 comp rx=0 tx=0).stop 2024-05-15T12:43:14.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.311+0000 7f35fd934640 10 monclient: shutdown 2024-05-15T12:43:14.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.311+0000 7f35fd934640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:14.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.311+0000 7f35fd934640 1 -- 172.21.15.117:0/3546014827 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f35f813c2a0 msgr2=0x7f35f813d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:14.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.311+0000 7f35fd934640 1 --2- 172.21.15.117:0/3546014827 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f35f813c2a0 0x7f35f813d8c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f35f000b9e0 tx=0x7f35f000bea0 comp rx=0 tx=0).stop 2024-05-15T12:43:14.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.312+0000 7f35fd934640 1 -- 172.21.15.117:0/3546014827 shutdown_connections 2024-05-15T12:43:14.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.312+0000 7f35fd934640 1 --2- 172.21.15.117:0/3546014827 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f35f813c2a0 0x7f35f813d8c0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:14.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.312+0000 7f35fd934640 1 --2- 172.21.15.117:0/3546014827 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f35d806ce60 0x7f35d806f320 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:14.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.312+0000 7f35fd934640 1 --2- 172.21.15.117:0/3546014827 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f35f80b4d00 0x7f35f813e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:14.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.312+0000 7f35fd934640 1 --2- 172.21.15.117:0/3546014827 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f35f80a94b0 0x7f35f81427a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:14.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.312+0000 7f35fd934640 1 -- 172.21.15.117:0/3546014827 >> 172.21.15.117:0/3546014827 conn(0x7f35f8066cc0 msgr2=0x7f35f80670a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:14.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.312+0000 7f35fd934640 1 -- 172.21.15.117:0/3546014827 shutdown_connections 2024-05-15T12:43:14.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.312+0000 7f35fd934640 1 -- 172.21.15.117:0/3546014827 wait complete. 2024-05-15T12:43:14.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.312+0000 7f35fd934640 1 librados: shutdown 2024-05-15T12:43:14.314 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378634 got 81604378634 for osd.4 2024-05-15T12:43:14.314 DEBUG:teuthology.parallel:result is None 2024-05-15T12:43:14.398 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.407+0000 7f3505cb0640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2024-05-15T12:43:14.398 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.408+0000 7f3505cb0640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-05-15T12:43:14.398 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.408+0000 7f3505cb0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:43:14.398 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.408+0000 7f3505cb0640 1 -- 172.21.15.117:0/3739604034 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f35001dbb80 con 0x7f35000affa0 2024-05-15T12:43:14.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.411+0000 7f34e7fff640 1 -- 172.21.15.117:0/3739604034 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f34f8059180 con 0x7f35000affa0 2024-05-15T12:43:14.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.411+0000 7f34e7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-05-15T12:43:14.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.411+0000 7f34e7fff640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:43:14.402 INFO:teuthology.orchestra.run.smithi117.stdout:81604378632 2024-05-15T12:43:14.404 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.413+0000 7f3505cb0640 10 librados: watch_flush enter 2024-05-15T12:43:14.404 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.413+0000 7f3505cb0640 10 librados: watch_flush exit 2024-05-15T12:43:14.404 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.413+0000 7f3505cb0640 20 client.4381.objecter shutdown clearing up homeless session... 2024-05-15T12:43:14.405 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.413+0000 7f3505cb0640 10 client.4381.objecter successfully canceled tick 2024-05-15T12:43:14.405 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.413+0000 7f3505cb0640 1 -- 172.21.15.117:0/3739604034 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f34ec06ceb0 msgr2=0x7f34ec06f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:14.405 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.413+0000 7f3505cb0640 1 --2- 172.21.15.117:0/3739604034 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f34ec06ceb0 0x7f34ec06f370 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f34f0036000 tx=0x7f34f0030000 comp rx=0 tx=0).stop 2024-05-15T12:43:14.405 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.413+0000 7f3505cb0640 10 monclient: shutdown 2024-05-15T12:43:14.405 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.413+0000 7f3505cb0640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:14.405 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.413+0000 7f3505cb0640 1 -- 172.21.15.117:0/3739604034 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f35000affa0 msgr2=0x7f35001dbef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:14.405 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.413+0000 7f3505cb0640 1 --2- 172.21.15.117:0/3739604034 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f35000affa0 0x7f35001dbef0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f34f800b9e0 tx=0x7f34f800bea0 comp rx=0 tx=0).stop 2024-05-15T12:43:14.405 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.414+0000 7f3505cb0640 1 -- 172.21.15.117:0/3739604034 shutdown_connections 2024-05-15T12:43:14.405 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.414+0000 7f3505cb0640 1 --2- 172.21.15.117:0/3739604034 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f34ec06ceb0 0x7f34ec06f370 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:14.405 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.414+0000 7f3505cb0640 1 --2- 172.21.15.117:0/3739604034 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f35000affa0 0x7f35001dbef0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:14.406 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.414+0000 7f3505cb0640 1 --2- 172.21.15.117:0/3739604034 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f35000b4d00 0x7f350013e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:14.406 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.414+0000 7f3505cb0640 1 --2- 172.21.15.117:0/3739604034 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f35000a94b0 0x7f3500142ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:14.406 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.414+0000 7f3505cb0640 1 -- 172.21.15.117:0/3739604034 >> 172.21.15.117:0/3739604034 conn(0x7f3500066cc0 msgr2=0x7f3500074610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:14.406 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.414+0000 7f3505cb0640 1 -- 172.21.15.117:0/3739604034 shutdown_connections 2024-05-15T12:43:14.406 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.414+0000 7f3505cb0640 1 -- 172.21.15.117:0/3739604034 wait complete. 2024-05-15T12:43:14.406 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.414+0000 7f3505cb0640 1 librados: shutdown 2024-05-15T12:43:14.415 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378632 got 81604378632 for osd.6 2024-05-15T12:43:14.415 DEBUG:teuthology.parallel:result is None 2024-05-15T12:43:14.415 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-05-15T12:43:14.415 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-15T12:43:14.513 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.523+0000 7fc6de4be640 10 monclient: get_monmap_and_config 2024-05-15T12:43:14.513 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.523+0000 7fc6de4be640 10 monclient: build_initial_monmap 2024-05-15T12:43:14.514 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.523+0000 7fc6de4be640 10 monclient: monmap: 2024-05-15T12:43:14.514 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:14.514 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:14.514 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:43:14.524657+0000 2024-05-15T12:43:14.514 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:43:14.524657+0000 2024-05-15T12:43:14.514 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:14.514 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:14.514 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T12:43:14.514 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T12:43:14.514 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T12:43:14.514 INFO:teuthology.orchestra.run.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T12:43:14.514 INFO:teuthology.orchestra.run.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T12:43:14.514 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:14.516 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.524+0000 7fc6de4be640 1 Processor -- start 2024-05-15T12:43:14.516 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.524+0000 7fc6de4be640 1 -- start start 2024-05-15T12:43:14.517 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.524+0000 7fc6de4be640 10 monclient: init 2024-05-15T12:43:14.517 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.524+0000 7fc6de4be640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:14.517 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.524+0000 7fc6de4be640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-05-15T12:43:14.517 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.525+0000 7fc6de4be640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fc6d806c900 0x7fc6d80aeab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:14.517 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.525+0000 7fc6de4be640 10 monclient(hunting): picked mon.noname-c con 0x7fc6d806c900 addr v2:172.21.15.178:3300/0 2024-05-15T12:43:14.517 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.525+0000 7fc6de4be640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc6d80aeff0 addr v1:172.21.15.117:6789/0 2024-05-15T12:43:14.517 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.525+0000 7fc6de4be640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc6d80af440 addr v1:172.21.15.178:6789/0 2024-05-15T12:43:14.517 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.525+0000 7fc6de4be640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6d8146ca0 con 0x7fc6d80aeff0 2024-05-15T12:43:14.517 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.525+0000 7fc6de4be640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6d80ae0d0 con 0x7fc6d80af440 2024-05-15T12:43:14.517 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.525+0000 7fc6de4be640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:14.517 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.525+0000 7fc6de4be640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6d806cce0 con 0x7fc6d806c900 2024-05-15T12:43:14.517 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.525+0000 7fc6de4be640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:14.517 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.525+0000 7fc6de4be640 10 monclient(hunting): authenticate will time out at 1357.766866s 2024-05-15T12:43:14.518 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.525+0000 7fc6d77fe640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fc6d80aeff0 0x7fc6d8145e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:48740/0 (socket says 172.21.15.117:48740) 2024-05-15T12:43:14.518 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.525+0000 7fc6d77fe640 1 -- 172.21.15.117:0/520749968 learned_addr learned my addr 172.21.15.117:0/520749968 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-05-15T12:43:14.518 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.525+0000 7fc6d7fff640 1 --2- 172.21.15.117:0/520749968 >> v2:172.21.15.178:3300/0 conn(0x7fc6d806c900 0x7fc6d80aeab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:14.518 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.526+0000 7fc6d7fff640 10 monclient(hunting): get_auth_request con 0x7fc6d806c900 auth_method 0 2024-05-15T12:43:14.518 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.526+0000 7fc6d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:14.518 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.526+0000 7fc6d7fff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:14.518 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.526+0000 7fc6d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:14.518 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.526+0000 7fc6d7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:14.518 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.526+0000 7fc6d7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:14.518 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.526+0000 7fc6d7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:14.519 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.528+0000 7fc6d6ffd640 1 -- 172.21.15.117:0/520749968 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 373691941 0 0) 0x7fc6d8146ca0 con 0x7fc6d80aeff0 2024-05-15T12:43:14.519 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.528+0000 7fc6d6ffd640 10 monclient(hunting): _negotiate 2024-05-15T12:43:14.519 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.528+0000 7fc6d6ffd640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:14.519 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.528+0000 7fc6d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:14.520 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.528+0000 7fc6d6ffd640 10 monclient(hunting): my global_id is 4371 2024-05-15T12:43:14.520 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.528+0000 7fc6d6ffd640 1 -- 172.21.15.117:0/520749968 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6c0003520 con 0x7fc6d80aeff0 2024-05-15T12:43:14.520 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.528+0000 7fc6d7fff640 10 monclient(hunting): handle_auth_done global_id 4384 payload 274 2024-05-15T12:43:14.520 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.528+0000 7fc6d7fff640 1 -- 172.21.15.117:0/520749968 >> v1:172.21.15.178:6789/0 conn(0x7fc6d80af440 legacy=0x7fc6d8146590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:43:14.520 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.528+0000 7fc6d7fff640 1 -- 172.21.15.117:0/520749968 >> v1:172.21.15.117:6789/0 conn(0x7fc6d80aeff0 legacy=0x7fc6d8145e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:14.520 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.528+0000 7fc6d7fff640 10 monclient: _finish_hunting 0 2024-05-15T12:43:14.520 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.528+0000 7fc6d7fff640 1 monclient: found mon.noname-c 2024-05-15T12:43:14.520 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.528+0000 7fc6d7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:14.520 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.528+0000 7fc6d7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.178:3300/0 2024-05-15T12:43:14.520 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.528+0000 7fc6d7fff640 1 -- 172.21.15.117:0/520749968 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6d80ae3b0 con 0x7fc6d806c900 2024-05-15T12:43:14.521 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.531+0000 7fc6d7fff640 1 --2- 172.21.15.117:0/520749968 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc6d806c900 0x7fc6d80aeab0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc6c80097a0 tx=0x7fc6c802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc6978628f1a9623 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:14.525 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.535+0000 7fc6d6ffd640 1 -- 172.21.15.117:0/520749968 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c80315a0 con 0x7fc6d806c900 2024-05-15T12:43:14.526 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.535+0000 7fc6d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:14.526 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.535+0000 7fc6d6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:43:14.526 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.535+0000 7fc6d6ffd640 10 monclient: dump: 2024-05-15T12:43:14.526 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:14.526 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:14.526 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:14.526 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:14.526 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:14.526 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:14.526 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:14.526 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:14.526 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:14.526 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:14.526 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.536+0000 7fc6d6ffd640 10 monclient: _finish_auth 0 2024-05-15T12:43:14.527 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.536+0000 7fc6d6ffd640 10 monclient: _check_auth_tickets 2024-05-15T12:43:14.527 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.536+0000 7fc6d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:14.527 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.536+0000 7fc6d6ffd640 1 -- 172.21.15.117:0/520749968 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6c8037070 con 0x7fc6d806c900 2024-05-15T12:43:14.527 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.536+0000 7fc6d6ffd640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:14.527 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.536+0000 7fc6d6ffd640 1 -- 172.21.15.117:0/520749968 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c8031960 con 0x7fc6d806c900 2024-05-15T12:43:14.527 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.536+0000 7fc6de4be640 5 monclient: authenticate success, global_id 4384 2024-05-15T12:43:14.527 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.536+0000 7fc6de4be640 10 monclient: get_monmap_and_config success 2024-05-15T12:43:14.527 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.536+0000 7fc6de4be640 4 set_mon_vals no callback set 2024-05-15T12:43:14.527 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.536+0000 7fc6d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:14.527 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.536+0000 7fc6d6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:43:14.527 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.536+0000 7fc6d6ffd640 10 monclient: dump: 2024-05-15T12:43:14.527 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:14.527 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:14.527 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:14.527 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:14.527 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:14.528 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:14.528 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:14.528 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:14.528 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:14.528 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:14.531 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.536+0000 7fc6de4be640 10 monclient: shutdown 2024-05-15T12:43:14.531 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.536+0000 7fc6de4be640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:14.531 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.536+0000 7fc6de4be640 1 -- 172.21.15.117:0/520749968 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc6d806c900 msgr2=0x7fc6d80aeab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:14.531 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.536+0000 7fc6de4be640 1 --2- 172.21.15.117:0/520749968 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc6d806c900 0x7fc6d80aeab0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc6c80097a0 tx=0x7fc6c802b840 comp rx=0 tx=0).stop 2024-05-15T12:43:14.532 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.536+0000 7fc6de4be640 1 -- 172.21.15.117:0/520749968 shutdown_connections 2024-05-15T12:43:14.532 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.536+0000 7fc6de4be640 1 --2- 172.21.15.117:0/520749968 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc6d806c900 0x7fc6d80aeab0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:14.532 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.536+0000 7fc6de4be640 1 -- 172.21.15.117:0/520749968 >> 172.21.15.117:0/520749968 conn(0x7fc6d80b2ac0 msgr2=0x7fc6d80b2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:14.532 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.536+0000 7fc6de4be640 1 -- 172.21.15.117:0/520749968 shutdown_connections 2024-05-15T12:43:14.532 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.536+0000 7fc6de4be640 1 -- 172.21.15.117:0/520749968 wait complete. 2024-05-15T12:43:14.532 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.537+0000 7fc6de4be640 10 monclient: build_initial_monmap 2024-05-15T12:43:14.532 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.537+0000 7fc6de4be640 10 monclient: monmap: 2024-05-15T12:43:14.532 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:14.532 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:43:14.532 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-05-15T12:43:14.532 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-05-15T12:43:14.532 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:14.532 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:14.532 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T12:43:14.533 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T12:43:14.533 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T12:43:14.533 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:14.533 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.537+0000 7fc6de4be640 1 librados: starting msgr at 2024-05-15T12:43:14.533 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.537+0000 7fc6de4be640 1 librados: starting objecter 2024-05-15T12:43:14.533 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.537+0000 7fc6de4be640 1 Processor -- start 2024-05-15T12:43:14.533 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.538+0000 7fc6de4be640 1 -- start start 2024-05-15T12:43:14.533 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.538+0000 7fc6de4be640 1 librados: setting wanted keys 2024-05-15T12:43:14.533 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.538+0000 7fc6de4be640 1 librados: calling monclient init 2024-05-15T12:43:14.533 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.538+0000 7fc6de4be640 10 monclient: init 2024-05-15T12:43:14.533 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.538+0000 7fc6de4be640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:14.533 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.538+0000 7fc6de4be640 10 monclient: _add_conns ranks=[2,1,0] 2024-05-15T12:43:14.533 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.538+0000 7fc6de4be640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc6d806c900 0x7fc6d81db620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:14.533 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.538+0000 7fc6de4be640 10 monclient(hunting): picked mon.noname-c con 0x7fc6d806c900 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:43:14.533 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.538+0000 7fc6de4be640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc6d80aeff0 0x7fc6d81487e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:14.534 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.538+0000 7fc6de4be640 10 monclient(hunting): picked mon.noname-b con 0x7fc6d80aeff0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:43:14.534 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.538+0000 7fc6de4be640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc6d80af440 0x7fc6d8148d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:14.534 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.538+0000 7fc6de4be640 10 monclient(hunting): picked mon.noname-a con 0x7fc6d80af440 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T12:43:14.534 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.538+0000 7fc6de4be640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:14.534 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.538+0000 7fc6de4be640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d806d730 con 0x7fc6d80af440 2024-05-15T12:43:14.534 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.538+0000 7fc6de4be640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:14.534 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.538+0000 7fc6de4be640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d814e440 con 0x7fc6d80aeff0 2024-05-15T12:43:14.534 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.538+0000 7fc6de4be640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:14.534 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.538+0000 7fc6de4be640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d814e5c0 con 0x7fc6d806c900 2024-05-15T12:43:14.534 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.538+0000 7fc6de4be640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:14.534 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.538+0000 7fc6de4be640 10 monclient(hunting): authenticate will time out at 1357.780451s 2024-05-15T12:43:14.534 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.539+0000 7fc6dca5b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc6d80af440 0x7fc6d8148d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:14.534 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.539+0000 7fc6d7fff640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc6d806c900 0x7fc6d81db620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:14.534 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.539+0000 7fc6d77fe640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc6d80aeff0 0x7fc6d81487e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:14.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.539+0000 7fc6d77fe640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc6d80aeff0 0x7fc6d81487e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:42704/0 (socket says 172.21.15.117:42704) 2024-05-15T12:43:14.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.539+0000 7fc6d77fe640 1 -- 172.21.15.117:0/1712918907 learned_addr learned my addr 172.21.15.117:0/1712918907 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:43:14.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.539+0000 7fc6d77fe640 10 monclient(hunting): get_auth_request con 0x7fc6d80aeff0 auth_method 0 2024-05-15T12:43:14.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.539+0000 7fc6d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:14.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.539+0000 7fc6d77fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:14.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.539+0000 7fc6d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:14.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.539+0000 7fc6dca5b640 10 monclient(hunting): get_auth_request con 0x7fc6d80af440 auth_method 0 2024-05-15T12:43:14.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.539+0000 7fc6dca5b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:14.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.539+0000 7fc6dca5b640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:14.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.539+0000 7fc6dca5b640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:14.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.539+0000 7fc6d7fff640 10 monclient(hunting): get_auth_request con 0x7fc6d806c900 auth_method 0 2024-05-15T12:43:14.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.539+0000 7fc6d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:14.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.539+0000 7fc6d7fff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:14.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.539+0000 7fc6d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:14.536 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.539+0000 7fc6d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:14.536 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.539+0000 7fc6d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:14.536 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.539+0000 7fc6d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:14.536 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.540+0000 7fc6d7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:14.536 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.540+0000 7fc6d7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:14.536 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.540+0000 7fc6d7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:14.536 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.540+0000 7fc6dca5b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:14.536 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.540+0000 7fc6dca5b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:14.536 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.540+0000 7fc6dca5b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:14.537 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.542+0000 7fc6d77fe640 10 monclient(hunting): handle_auth_done global_id 4387 payload 824 2024-05-15T12:43:14.537 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.542+0000 7fc6d77fe640 1 -- 172.21.15.117:0/1712918907 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc6d806c900 msgr2=0x7fc6d81db620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:14.537 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.542+0000 7fc6d77fe640 1 --2- 172.21.15.117:0/1712918907 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc6d806c900 0x7fc6d81db620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:14.537 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.542+0000 7fc6d77fe640 1 -- 172.21.15.117:0/1712918907 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc6d80af440 msgr2=0x7fc6d8148d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:14.538 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.542+0000 7fc6d77fe640 1 --2- 172.21.15.117:0/1712918907 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc6d80af440 0x7fc6d8148d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:14.538 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.542+0000 7fc6d77fe640 10 monclient: _finish_hunting 0 2024-05-15T12:43:14.538 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.542+0000 7fc6d77fe640 1 monclient: found mon.noname-b 2024-05-15T12:43:14.538 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.542+0000 7fc6d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:14.538 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.542+0000 7fc6d77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2024-05-15T12:43:14.538 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.542+0000 7fc6d77fe640 1 -- 172.21.15.117:0/1712918907 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6d814a1a0 con 0x7fc6d80aeff0 2024-05-15T12:43:14.538 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.544+0000 7fc6d77fe640 1 --2- 172.21.15.117:0/1712918907 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc6d80aeff0 0x7fc6d81487e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc00c430 tx=0x7fc6cc00c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:14.538 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.544+0000 7fc6d4ff9640 10 client.?.objecter ms_handle_connect 0x7fc6d80aeff0 2024-05-15T12:43:14.538 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.544+0000 7fc6d4ff9640 10 client.?.objecter resend_mon_ops 2024-05-15T12:43:14.539 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.548+0000 7fc6d4ff9640 1 -- 172.21.15.117:0/1712918907 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6cc017070 con 0x7fc6d80aeff0 2024-05-15T12:43:14.539 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.548+0000 7fc6d4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:14.540 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.548+0000 7fc6d4ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:43:14.540 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.548+0000 7fc6d4ff9640 10 monclient: dump: 2024-05-15T12:43:14.540 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:14.540 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:14.540 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:14.540 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:14.540 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:14.540 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:14.540 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:14.540 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:14.540 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:14.540 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:14.540 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.548+0000 7fc6d4ff9640 10 monclient: _finish_auth 0 2024-05-15T12:43:14.540 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.548+0000 7fc6d4ff9640 10 monclient: _check_auth_tickets 2024-05-15T12:43:14.540 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.548+0000 7fc6d4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:14.540 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.548+0000 7fc6d4ff9640 1 -- 172.21.15.117:0/1712918907 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6cc005ce0 con 0x7fc6d80aeff0 2024-05-15T12:43:14.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.548+0000 7fc6d4ff9640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:14.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.548+0000 7fc6d4ff9640 1 -- 172.21.15.117:0/1712918907 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6cc0072f0 con 0x7fc6d80aeff0 2024-05-15T12:43:14.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.548+0000 7fc6de4be640 5 monclient: authenticate success, global_id 4387 2024-05-15T12:43:14.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.548+0000 7fc6de4be640 10 monclient: _renew_subs 2024-05-15T12:43:14.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.548+0000 7fc6de4be640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:43:14.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.548+0000 7fc6de4be640 1 -- 172.21.15.117:0/1712918907 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6d8149260 con 0x7fc6d80aeff0 2024-05-15T12:43:14.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.548+0000 7fc6d4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:14.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.548+0000 7fc6d4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:43:14.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.548+0000 7fc6d4ff9640 10 monclient: dump: 2024-05-15T12:43:14.541 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:14.541 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:14.541 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:14.541 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:14.541 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:14.541 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:14.541 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:14.542 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:14.542 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:14.542 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:14.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.548+0000 7fc6d5ffb640 4 set_mon_vals no callback set 2024-05-15T12:43:14.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.548+0000 7fc6de4be640 10 client.4387.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:43:14.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.549+0000 7fc6de4be640 10 monclient: _renew_subs 2024-05-15T12:43:14.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.549+0000 7fc6de4be640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:43:14.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.549+0000 7fc6de4be640 1 -- 172.21.15.117:0/1712918907 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6d81dbd40 con 0x7fc6d80aeff0 2024-05-15T12:43:14.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.549+0000 7fc6de4be640 1 librados: init done 2024-05-15T12:43:14.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.549+0000 7fc6de4be640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:14.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.549+0000 7fc6de4be640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:14.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.549+0000 7fc6de4be640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:43:14.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.549+0000 7fc6de4be640 1 -- 172.21.15.117:0/1712918907 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6d81daf20 con 0x7fc6d80aeff0 2024-05-15T12:43:14.557 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.567+0000 7fc6d4ff9640 1 -- 172.21.15.117:0/1712918907 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fc6cc01a070 con 0x7fc6d80aeff0 2024-05-15T12:43:14.558 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.567+0000 7fc6d4ff9640 1 --2- 172.21.15.117:0/1712918907 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fc6c406ceb0 0x7fc6c406f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:14.558 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.568+0000 7fc6d7fff640 1 --2- 172.21.15.117:0/1712918907 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fc6c406ceb0 0x7fc6c406f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:14.558 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.568+0000 7fc6d7fff640 10 monclient: get_auth_request con 0x7fc6c406ceb0 auth_method 0 2024-05-15T12:43:14.558 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.568+0000 7fc6d7fff640 1 --2- 172.21.15.117:0/1712918907 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fc6c406ceb0 0x7fc6c406f370 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc6c8036000 tx=0x7fc6c803d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:14.558 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.568+0000 7fc6d4ff9640 10 client.4387.objecter ms_handle_connect 0x7fc6c406ceb0 2024-05-15T12:43:14.560 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.570+0000 7fc6d4ff9640 1 -- 172.21.15.117:0/1712918907 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fc6cc056300 con 0x7fc6d80aeff0 2024-05-15T12:43:14.560 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.570+0000 7fc6d4ff9640 10 client.4387.objecter ms_dispatch 0x7fc6d8000b90 osd_map(26..26 src has 1..26) v4 2024-05-15T12:43:14.560 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.570+0000 7fc6d4ff9640 3 client.4387.objecter handle_osd_map got epochs [26,26] > 0 2024-05-15T12:43:14.560 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.570+0000 7fc6d4ff9640 3 client.4387.objecter handle_osd_map decoding full epoch 26 2024-05-15T12:43:14.561 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.570+0000 7fc6d4ff9640 20 client.4387.objecter dump_active .. 0 homeless 2024-05-15T12:43:14.796 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.805+0000 7fc6d4ff9640 1 -- 172.21.15.117:0/1712918907 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fc6cc021020 con 0x7fc6d80aeff0 2024-05-15T12:43:14.796 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.806+0000 7fc6d4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:14.796 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.806+0000 7fc6d4ff9640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:43:14.964 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.974+0000 7fc6de4be640 1 -- 172.21.15.117:0/1712918907 --> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc6d81db280 con 0x7fc6c406ceb0 2024-05-15T12:43:14.966 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.976+0000 7fc6d4ff9640 1 -- 172.21.15.117:0/1712918907 <== mgr.4107 v2:172.21.15.117:6800/1785766523 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+35506 (secure 0 0 0) 0x7fc6d81db280 con 0x7fc6c406ceb0 2024-05-15T12:43:14.966 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:43:14.967 INFO:teuthology.orchestra.run.smithi117.stderr:dumped all 2024-05-15T12:43:14.968 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.977+0000 7fc6de4be640 10 librados: watch_flush enter 2024-05-15T12:43:14.968 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.977+0000 7fc6de4be640 10 librados: watch_flush exit 2024-05-15T12:43:14.968 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.977+0000 7fc6de4be640 20 client.4387.objecter shutdown clearing up homeless session... 2024-05-15T12:43:14.968 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.977+0000 7fc6de4be640 10 client.4387.objecter successfully canceled tick 2024-05-15T12:43:14.968 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.978+0000 7fc6de4be640 1 -- 172.21.15.117:0/1712918907 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fc6c406ceb0 msgr2=0x7fc6c406f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:14.969 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.978+0000 7fc6de4be640 1 --2- 172.21.15.117:0/1712918907 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fc6c406ceb0 0x7fc6c406f370 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc6c8036000 tx=0x7fc6c803d040 comp rx=0 tx=0).stop 2024-05-15T12:43:14.969 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.978+0000 7fc6de4be640 10 monclient: shutdown 2024-05-15T12:43:14.969 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.978+0000 7fc6de4be640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:14.969 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.978+0000 7fc6de4be640 1 -- 172.21.15.117:0/1712918907 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc6d80aeff0 msgr2=0x7fc6d81487e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:14.969 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.978+0000 7fc6de4be640 1 --2- 172.21.15.117:0/1712918907 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc6d80aeff0 0x7fc6d81487e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc00c430 tx=0x7fc6cc00c8f0 comp rx=0 tx=0).stop 2024-05-15T12:43:14.969 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.978+0000 7fc6de4be640 1 -- 172.21.15.117:0/1712918907 shutdown_connections 2024-05-15T12:43:14.969 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.978+0000 7fc6de4be640 1 --2- 172.21.15.117:0/1712918907 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc6d80af440 0x7fc6d8148d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:14.969 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.978+0000 7fc6de4be640 1 --2- 172.21.15.117:0/1712918907 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fc6c406ceb0 0x7fc6c406f370 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:14.969 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.978+0000 7fc6de4be640 1 --2- 172.21.15.117:0/1712918907 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc6d80aeff0 0x7fc6d81487e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:14.969 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.978+0000 7fc6de4be640 1 --2- 172.21.15.117:0/1712918907 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc6d806c900 0x7fc6d81db620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:14.969 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.978+0000 7fc6de4be640 1 -- 172.21.15.117:0/1712918907 >> 172.21.15.117:0/1712918907 conn(0x7fc6d80b2ac0 msgr2=0x7fc6d80b2ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:14.969 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.978+0000 7fc6de4be640 1 -- 172.21.15.117:0/1712918907 shutdown_connections 2024-05-15T12:43:14.969 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.978+0000 7fc6de4be640 1 -- 172.21.15.117:0/1712918907 wait complete. 2024-05-15T12:43:14.969 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:14.978+0000 7fc6de4be640 1 librados: shutdown 2024-05-15T12:43:14.979 INFO:teuthology.orchestra.run.smithi117.stdout:{"pg_ready":true,"pg_map":{"version":42,"stamp":"2024-05-15T12:43:13.910512+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218832,"kb_used_data":2664,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755888,"statfs":{"total":773094113280,"available":772870029312,"internally_reserved":0,"allocated":2727936,"data_stored":1468512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001128"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-15T12:43:10.884536+0000","last_change":"2024-05-15T12:43:09.429828+0000","last_active":"2024-05-15T12:43:10.884536+0000","last_peered":"2024-05-15T12:43:10.884536+0000","last_clean":"2024-05-15T12:43:10.884536+0000","last_became_active":"2024-05-15T12:43:09.424460+0000","last_became_peered":"2024-05-15T12:43:09.424460+0000","last_unstale":"2024-05-15T12:43:10.884536+0000","last_undegraded":"2024-05-15T12:43:10.884536+0000","last_fullsized":"2024-05-15T12:43:10.884536+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_clean_scrub_stamp":"2024-05-15T12:43:07.133352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:12:21.424590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-15T12:43:10.812177+0000","last_change":"2024-05-15T12:43:09.478046+0000","last_active":"2024-05-15T12:43:10.812177+0000","last_peered":"2024-05-15T12:43:10.812177+0000","last_clean":"2024-05-15T12:43:10.812177+0000","last_became_active":"2024-05-15T12:43:09.448237+0000","last_became_peered":"2024-05-15T12:43:09.448237+0000","last_unstale":"2024-05-15T12:43:10.812177+0000","last_undegraded":"2024-05-15T12:43:10.812177+0000","last_fullsized":"2024-05-15T12:43:10.812177+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_clean_scrub_stamp":"2024-05-15T12:43:07.133352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:11:17.055288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-15T12:43:10.878936+0000","last_change":"2024-05-15T12:43:08.778833+0000","last_active":"2024-05-15T12:43:10.878936+0000","last_peered":"2024-05-15T12:43:10.878936+0000","last_clean":"2024-05-15T12:43:10.878936+0000","last_became_active":"2024-05-15T12:43:08.773614+0000","last_became_peered":"2024-05-15T12:43:08.773614+0000","last_unstale":"2024-05-15T12:43:10.878936+0000","last_undegraded":"2024-05-15T12:43:10.878936+0000","last_fullsized":"2024-05-15T12:43:10.878936+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_clean_scrub_stamp":"2024-05-15T12:43:07.133352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:22:59.785687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-15T12:43:10.806149+0000","last_change":"2024-05-15T12:43:09.384765+0000","last_active":"2024-05-15T12:43:10.806149+0000","last_peered":"2024-05-15T12:43:10.806149+0000","last_clean":"2024-05-15T12:43:10.806149+0000","last_became_active":"2024-05-15T12:43:09.377725+0000","last_became_peered":"2024-05-15T12:43:09.377725+0000","last_unstale":"2024-05-15T12:43:10.806149+0000","last_undegraded":"2024-05-15T12:43:10.806149+0000","last_fullsized":"2024-05-15T12:43:10.806149+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_clean_scrub_stamp":"2024-05-15T12:43:07.133352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:57:59.671602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-15T12:43:10.861367+0000","last_change":"2024-05-15T12:43:09.375465+0000","last_active":"2024-05-15T12:43:10.861367+0000","last_peered":"2024-05-15T12:43:10.861367+0000","last_clean":"2024-05-15T12:43:10.861367+0000","last_became_active":"2024-05-15T12:43:09.363876+0000","last_became_peered":"2024-05-15T12:43:09.363876+0000","last_unstale":"2024-05-15T12:43:10.861367+0000","last_undegraded":"2024-05-15T12:43:10.861367+0000","last_fullsized":"2024-05-15T12:43:10.861367+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_clean_scrub_stamp":"2024-05-15T12:43:07.133352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:27:46.599385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-05-15T12:43:10.054499+0000","last_change":"2024-05-15T12:43:08.634999+0000","last_active":"2024-05-15T12:43:10.054499+0000","last_peered":"2024-05-15T12:43:10.054499+0000","last_clean":"2024-05-15T12:43:10.054499+0000","last_became_active":"2024-05-15T12:43:08.600745+0000","last_became_peered":"2024-05-15T12:43:08.600745+0000","last_unstale":"2024-05-15T12:43:10.054499+0000","last_undegraded":"2024-05-15T12:43:10.054499+0000","last_fullsized":"2024-05-15T12:43:10.054499+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_clean_scrub_stamp":"2024-05-15T12:43:07.133352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T15:01:15.586689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-15T12:43:10.837850+0000","last_change":"2024-05-15T12:43:08.704425+0000","last_active":"2024-05-15T12:43:10.837850+0000","last_peered":"2024-05-15T12:43:10.837850+0000","last_clean":"2024-05-15T12:43:10.837850+0000","last_became_active":"2024-05-15T12:43:08.677751+0000","last_became_peered":"2024-05-15T12:43:08.677751+0000","last_unstale":"2024-05-15T12:43:10.837850+0000","last_undegraded":"2024-05-15T12:43:10.837850+0000","last_fullsized":"2024-05-15T12:43:10.837850+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_clean_scrub_stamp":"2024-05-15T12:43:07.133352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-17T00:22:51.843061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-15T12:43:10.873093+0000","last_change":"2024-05-15T12:43:09.453018+0000","last_active":"2024-05-15T12:43:10.873093+0000","last_peered":"2024-05-15T12:43:10.873093+0000","last_clean":"2024-05-15T12:43:10.873093+0000","last_became_active":"2024-05-15T12:43:09.445878+0000","last_became_peered":"2024-05-15T12:43:09.445878+0000","last_unstale":"2024-05-15T12:43:10.873093+0000","last_undegraded":"2024-05-15T12:43:10.873093+0000","last_fullsized":"2024-05-15T12:43:10.873093+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_clean_scrub_stamp":"2024-05-15T12:43:07.133352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:55:05.766298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'76","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-15T12:43:11.944076+0000","last_change":"2024-05-15T12:43:04.763767+0000","last_active":"2024-05-15T12:43:11.944076+0000","last_peered":"2024-05-15T12:43:11.944076+0000","last_clean":"2024-05-15T12:43:11.944076+0000","last_became_active":"2024-05-15T12:43:04.148735+0000","last_became_peered":"2024-05-15T12:43:04.148735+0000","last_unstale":"2024-05-15T12:43:11.944076+0000","last_undegraded":"2024-05-15T12:43:11.944076+0000","last_fullsized":"2024-05-15T12:43:11.944076+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:42:53.071197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:42:53.071197+0000","last_clean_scrub_stamp":"2024-05-15T12:42:53.071197+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:45:23.691461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":20,"seq":85899345930,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27656,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344184,"statfs":{"total":96636764160,"available":96608444416,"internally_reserved":0,"allocated":651264,"data_stored":486255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":20,"seq":85899345932,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":237568,"data_stored":82667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":20,"seq":85899345931,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":237568,"data_stored":82667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":19,"seq":81604378632,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":237568,"data_stored":82667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":19,"seq":81604378634,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":237568,"data_stored":82667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509448,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":249856,"data_stored":88415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542151,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27644,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344196,"statfs":{"total":96636764160,"available":96608456704,"internally_reserved":0,"allocated":638976,"data_stored":480507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":237568,"data_stored":82667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T12:43:14.980 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-15T12:43:15.087 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.097+0000 7f4a5a3fe640 10 monclient: get_monmap_and_config 2024-05-15T12:43:15.087 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.097+0000 7f4a5a3fe640 10 monclient: build_initial_monmap 2024-05-15T12:43:15.087 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.097+0000 7f4a5a3fe640 10 monclient: monmap: 2024-05-15T12:43:15.087 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:15.087 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:15.087 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:43:15.098098+0000 2024-05-15T12:43:15.087 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:43:15.098098+0000 2024-05-15T12:43:15.087 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:15.087 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:15.088 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T12:43:15.088 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T12:43:15.088 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T12:43:15.088 INFO:teuthology.orchestra.run.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T12:43:15.088 INFO:teuthology.orchestra.run.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T12:43:15.088 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:15.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.097+0000 7f4a5a3fe640 1 Processor -- start 2024-05-15T12:43:15.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.097+0000 7f4a5a3fe640 1 -- start start 2024-05-15T12:43:15.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.097+0000 7f4a5a3fe640 10 monclient: init 2024-05-15T12:43:15.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.098+0000 7f4a5a3fe640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:15.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.098+0000 7f4a5a3fe640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-05-15T12:43:15.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.098+0000 7f4a5a3fe640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4a5406c7e0 0x7f4a540ae990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:15.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.098+0000 7f4a5a3fe640 10 monclient(hunting): picked mon.noname-b con 0x7f4a5406c7e0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:43:15.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.098+0000 7f4a5a3fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4a5406cbc0 addr v1:172.21.15.178:6789/0 2024-05-15T12:43:15.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.098+0000 7f4a5a3fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4a54142bf0 addr v1:172.21.15.117:6789/0 2024-05-15T12:43:15.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.098+0000 7f4a5a3fe640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a5406da60 con 0x7f4a54142bf0 2024-05-15T12:43:15.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.098+0000 7f4a5a3fe640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a540b0360 con 0x7f4a5406cbc0 2024-05-15T12:43:15.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.098+0000 7f4a5a3fe640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:15.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.098+0000 7f4a5a3fe640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a540adfb0 con 0x7f4a5406c7e0 2024-05-15T12:43:15.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.098+0000 7f4a5a3fe640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:15.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.098+0000 7f4a5a3fe640 10 monclient(hunting): authenticate will time out at 1358.340461s 2024-05-15T12:43:15.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.102+0000 7f4a53fff640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4a5406c7e0 0x7f4a540ae990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:15.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.102+0000 7f4a537fe640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f4a5406cbc0 0x7f4a540aeed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.117:54024/0 (socket says 172.21.15.117:54024) 2024-05-15T12:43:15.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.102+0000 7f4a537fe640 1 -- 172.21.15.117:0/3781372812 learned_addr learned my addr 172.21.15.117:0/3781372812 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-05-15T12:43:15.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.102+0000 7f4a53fff640 10 monclient(hunting): get_auth_request con 0x7f4a5406c7e0 auth_method 0 2024-05-15T12:43:15.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.102+0000 7f4a53fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:15.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.102+0000 7f4a53fff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:15.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.102+0000 7f4a53fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:15.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.103+0000 7f4a53fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:15.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.103+0000 7f4a53fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:15.094 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.103+0000 7f4a53fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:15.095 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.105+0000 7f4a53fff640 10 monclient(hunting): handle_auth_done global_id 4322 payload 274 2024-05-15T12:43:15.095 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.105+0000 7f4a53fff640 1 -- 172.21.15.117:0/3781372812 >> v1:172.21.15.178:6789/0 conn(0x7f4a5406cbc0 legacy=0x7f4a540aeed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:15.095 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.105+0000 7f4a53fff640 1 -- 172.21.15.117:0/3781372812 >> v1:172.21.15.117:6789/0 conn(0x7f4a54142bf0 legacy=0x7f4a54143060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:43:15.096 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.105+0000 7f4a53fff640 10 monclient: _finish_hunting 0 2024-05-15T12:43:15.096 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.105+0000 7f4a53fff640 1 monclient: found mon.noname-b 2024-05-15T12:43:15.096 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.105+0000 7f4a53fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:15.096 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.105+0000 7f4a53fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.117:3301/0 2024-05-15T12:43:15.096 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.105+0000 7f4a53fff640 1 -- 172.21.15.117:0/3781372812 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a540ae290 con 0x7f4a5406c7e0 2024-05-15T12:43:15.097 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.107+0000 7f4a53fff640 1 --2- 172.21.15.117:0/3781372812 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4a5406c7e0 0x7f4a540ae990 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4a440097d0 tx=0x7f4a4402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5b0c83b9b93aef8 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:15.151 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.160+0000 7f4a52ffd640 1 -- 172.21.15.117:0/3781372812 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a44035020 con 0x7f4a5406c7e0 2024-05-15T12:43:15.151 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.160+0000 7f4a52ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:15.151 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.160+0000 7f4a52ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:43:15.151 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.160+0000 7f4a52ffd640 10 monclient: dump: 2024-05-15T12:43:15.151 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:15.151 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:15.152 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:15.152 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:15.152 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:15.152 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:15.152 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:15.152 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:15.152 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:15.152 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:15.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.160+0000 7f4a52ffd640 10 monclient: _finish_auth 0 2024-05-15T12:43:15.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.160+0000 7f4a52ffd640 10 monclient: _check_auth_tickets 2024-05-15T12:43:15.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.160+0000 7f4a52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:15.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.160+0000 7f4a52ffd640 1 -- 172.21.15.117:0/3781372812 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a44031800 con 0x7f4a5406c7e0 2024-05-15T12:43:15.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.160+0000 7f4a5a3fe640 5 monclient: authenticate success, global_id 4322 2024-05-15T12:43:15.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.160+0000 7f4a5a3fe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:43:15.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.160+0000 7f4a52ffd640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:15.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.160+0000 7f4a5a3fe640 10 monclient: get_monmap_and_config success 2024-05-15T12:43:15.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.160+0000 7f4a52ffd640 1 -- 172.21.15.117:0/3781372812 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a44031bd0 con 0x7f4a5406c7e0 2024-05-15T12:43:15.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.160+0000 7f4a52ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:15.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.160+0000 7f4a5a3fe640 4 set_mon_vals no callback set 2024-05-15T12:43:15.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.160+0000 7f4a52ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T12:43:15.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.160+0000 7f4a52ffd640 10 monclient: dump: 2024-05-15T12:43:15.153 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:15.153 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:15.153 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:15.153 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:15.153 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:15.153 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:15.153 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:15.153 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:15.153 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:15.154 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:15.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.161+0000 7f4a5a3fe640 10 monclient: shutdown 2024-05-15T12:43:15.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.161+0000 7f4a5a3fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:15.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.161+0000 7f4a5a3fe640 1 -- 172.21.15.117:0/3781372812 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4a5406c7e0 msgr2=0x7f4a540ae990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:15.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.161+0000 7f4a5a3fe640 1 --2- 172.21.15.117:0/3781372812 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4a5406c7e0 0x7f4a540ae990 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4a440097d0 tx=0x7f4a4402b8a0 comp rx=0 tx=0).stop 2024-05-15T12:43:15.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.161+0000 7f4a5a3fe640 1 -- 172.21.15.117:0/3781372812 shutdown_connections 2024-05-15T12:43:15.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.161+0000 7f4a5a3fe640 1 --2- 172.21.15.117:0/3781372812 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4a5406c7e0 0x7f4a540ae990 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:15.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.161+0000 7f4a5a3fe640 1 -- 172.21.15.117:0/3781372812 >> 172.21.15.117:0/3781372812 conn(0x7f4a540b5f40 msgr2=0x7f4a540b6350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:15.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.161+0000 7f4a5a3fe640 1 -- 172.21.15.117:0/3781372812 shutdown_connections 2024-05-15T12:43:15.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.161+0000 7f4a5a3fe640 1 -- 172.21.15.117:0/3781372812 wait complete. 2024-05-15T12:43:15.157 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.161+0000 7f4a5a3fe640 10 monclient: build_initial_monmap 2024-05-15T12:43:15.157 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.161+0000 7f4a5a3fe640 10 monclient: monmap: 2024-05-15T12:43:15.157 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:15.157 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:43:15.158 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-05-15T12:43:15.158 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-05-15T12:43:15.158 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:15.158 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:15.158 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T12:43:15.158 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T12:43:15.158 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T12:43:15.158 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:15.158 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.162+0000 7f4a5a3fe640 1 librados: starting msgr at 2024-05-15T12:43:15.158 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.162+0000 7f4a5a3fe640 1 librados: starting objecter 2024-05-15T12:43:15.158 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.162+0000 7f4a5a3fe640 1 Processor -- start 2024-05-15T12:43:15.158 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.162+0000 7f4a5a3fe640 1 -- start start 2024-05-15T12:43:15.158 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.162+0000 7f4a5a3fe640 1 librados: setting wanted keys 2024-05-15T12:43:15.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.162+0000 7f4a5a3fe640 1 librados: calling monclient init 2024-05-15T12:43:15.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.162+0000 7f4a5a3fe640 10 monclient: init 2024-05-15T12:43:15.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.163+0000 7f4a5a3fe640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:15.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.163+0000 7f4a5a3fe640 10 monclient: _add_conns ranks=[0,1,2] 2024-05-15T12:43:15.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.163+0000 7f4a5a3fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4a5406c7e0 0x7f4a541446b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:15.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.163+0000 7f4a5a3fe640 10 monclient(hunting): picked mon.noname-a con 0x7f4a5406c7e0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T12:43:15.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.163+0000 7f4a5a3fe640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4a5406cbc0 0x7f4a54144120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:15.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.163+0000 7f4a5a3fe640 10 monclient(hunting): picked mon.noname-b con 0x7f4a5406cbc0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:43:15.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.163+0000 7f4a5a3fe640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4a54142bf0 0x7f4a541401b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:15.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.163+0000 7f4a5a3fe640 10 monclient(hunting): picked mon.noname-c con 0x7f4a54142bf0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:43:15.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.163+0000 7f4a5a3fe640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:15.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.163+0000 7f4a5a3fe640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a541438f0 con 0x7f4a5406c7e0 2024-05-15T12:43:15.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.163+0000 7f4a5a3fe640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:15.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.163+0000 7f4a5a3fe640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a54143770 con 0x7f4a5406cbc0 2024-05-15T12:43:15.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.163+0000 7f4a5a3fe640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:15.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.163+0000 7f4a5a3fe640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a54146510 con 0x7f4a54142bf0 2024-05-15T12:43:15.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.163+0000 7f4a5a3fe640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:15.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.163+0000 7f4a5a3fe640 10 monclient(hunting): authenticate will time out at 1358.404914s 2024-05-15T12:43:15.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.164+0000 7f4a53fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4a5406c7e0 0x7f4a541446b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:15.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.165+0000 7f4a53fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4a5406c7e0 0x7f4a541446b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:60436/0 (socket says 172.21.15.117:60436) 2024-05-15T12:43:15.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.165+0000 7f4a53fff640 1 -- 172.21.15.117:0/3996711144 learned_addr learned my addr 172.21.15.117:0/3996711144 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:43:15.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.165+0000 7f4a53fff640 10 monclient(hunting): get_auth_request con 0x7f4a5406c7e0 auth_method 0 2024-05-15T12:43:15.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.165+0000 7f4a53fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:15.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.165+0000 7f4a53fff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:15.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.165+0000 7f4a53fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:15.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.166+0000 7f4a537fe640 1 --2- 172.21.15.117:0/3996711144 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4a5406cbc0 0x7f4a54144120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:15.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.166+0000 7f4a537fe640 10 monclient(hunting): get_auth_request con 0x7f4a5406cbc0 auth_method 0 2024-05-15T12:43:15.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.166+0000 7f4a537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:15.161 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.166+0000 7f4a537fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:15.161 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.166+0000 7f4a537fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:15.161 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.166+0000 7f4a53fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:15.161 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.166+0000 7f4a53fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:15.161 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.166+0000 7f4a5899b640 1 --2- 172.21.15.117:0/3996711144 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4a54142bf0 0x7f4a541401b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:15.161 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.166+0000 7f4a53fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:15.161 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.166+0000 7f4a537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:15.161 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.166+0000 7f4a537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:15.161 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.166+0000 7f4a537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:15.161 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.167+0000 7f4a5899b640 10 monclient(hunting): get_auth_request con 0x7f4a54142bf0 auth_method 0 2024-05-15T12:43:15.161 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.167+0000 7f4a5899b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:15.161 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.167+0000 7f4a5899b640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:15.161 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.167+0000 7f4a5899b640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:15.163 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.168+0000 7f4a5899b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:15.163 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.168+0000 7f4a5899b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:15.163 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.168+0000 7f4a5899b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:15.163 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.169+0000 7f4a53fff640 10 monclient(hunting): handle_auth_done global_id 4377 payload 824 2024-05-15T12:43:15.163 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.169+0000 7f4a53fff640 1 -- 172.21.15.117:0/3996711144 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4a54142bf0 msgr2=0x7f4a541401b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:15.163 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.169+0000 7f4a53fff640 1 --2- 172.21.15.117:0/3996711144 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4a54142bf0 0x7f4a541401b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:15.163 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.169+0000 7f4a53fff640 1 -- 172.21.15.117:0/3996711144 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4a5406cbc0 msgr2=0x7f4a54144120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:15.163 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.169+0000 7f4a53fff640 1 --2- 172.21.15.117:0/3996711144 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4a5406cbc0 0x7f4a54144120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:15.163 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.169+0000 7f4a53fff640 10 monclient: _finish_hunting 0 2024-05-15T12:43:15.163 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.169+0000 7f4a53fff640 1 monclient: found mon.noname-a 2024-05-15T12:43:15.163 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.169+0000 7f4a53fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:15.163 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.169+0000 7f4a53fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.117:3300/0 2024-05-15T12:43:15.163 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.169+0000 7f4a53fff640 1 -- 172.21.15.117:0/3996711144 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a54146230 con 0x7f4a5406c7e0 2024-05-15T12:43:15.164 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.173+0000 7f4a53fff640 1 --2- 172.21.15.117:0/3996711144 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4a5406c7e0 0x7f4a541446b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4a440097d0 tx=0x7f4a440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:15.164 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.173+0000 7f4a50ff9640 10 client.?.objecter ms_handle_connect 0x7f4a5406c7e0 2024-05-15T12:43:15.164 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.173+0000 7f4a50ff9640 10 client.?.objecter resend_mon_ops 2024-05-15T12:43:15.168 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.177+0000 7f4a50ff9640 1 -- 172.21.15.117:0/3996711144 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a4403bd70 con 0x7f4a5406c7e0 2024-05-15T12:43:15.168 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.177+0000 7f4a50ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:15.168 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.177+0000 7f4a50ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T12:43:15.168 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.177+0000 7f4a50ff9640 10 monclient: dump: 2024-05-15T12:43:15.168 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:15.168 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:15.169 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:15.169 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:15.169 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:15.169 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:15.169 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:15.169 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:15.169 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:15.169 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:15.169 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.177+0000 7f4a50ff9640 10 monclient: _finish_auth 0 2024-05-15T12:43:15.169 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.177+0000 7f4a50ff9640 10 monclient: _check_auth_tickets 2024-05-15T12:43:15.169 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.177+0000 7f4a50ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:15.169 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.177+0000 7f4a50ff9640 1 -- 172.21.15.117:0/3996711144 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a440453f0 con 0x7f4a5406c7e0 2024-05-15T12:43:15.169 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.177+0000 7f4a5a3fe640 5 monclient: authenticate success, global_id 4377 2024-05-15T12:43:15.169 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.177+0000 7f4a5a3fe640 10 monclient: _renew_subs 2024-05-15T12:43:15.169 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.177+0000 7f4a5a3fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:43:15.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.177+0000 7f4a5a3fe640 1 -- 172.21.15.117:0/3996711144 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a54144cb0 con 0x7f4a5406c7e0 2024-05-15T12:43:15.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.177+0000 7f4a50ff9640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:15.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.177+0000 7f4a50ff9640 1 -- 172.21.15.117:0/3996711144 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a440457e0 con 0x7f4a5406c7e0 2024-05-15T12:43:15.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.177+0000 7f4a50ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:15.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.177+0000 7f4a50ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T12:43:15.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.177+0000 7f4a50ff9640 10 monclient: dump: 2024-05-15T12:43:15.170 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:15.170 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:15.170 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:15.170 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:15.170 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:15.170 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:15.170 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:15.171 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:15.171 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:15.171 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:15.171 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.177+0000 7f4a527fc640 4 set_mon_vals no callback set 2024-05-15T12:43:15.171 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.177+0000 7f4a5a3fe640 10 client.4377.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:43:15.171 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.177+0000 7f4a5a3fe640 10 monclient: _renew_subs 2024-05-15T12:43:15.171 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.177+0000 7f4a5a3fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:43:15.171 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.177+0000 7f4a5a3fe640 1 -- 172.21.15.117:0/3996711144 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a541406f0 con 0x7f4a5406c7e0 2024-05-15T12:43:15.171 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.177+0000 7f4a5a3fe640 1 librados: init done 2024-05-15T12:43:15.171 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.178+0000 7f4a5a3fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:15.171 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.178+0000 7f4a5a3fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:15.171 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.178+0000 7f4a5a3fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:43:15.171 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.178+0000 7f4a5a3fe640 1 -- 172.21.15.117:0/3996711144 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a5404f0e0 con 0x7f4a5406c7e0 2024-05-15T12:43:15.466 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.476+0000 7f4a50ff9640 1 -- 172.21.15.117:0/3996711144 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f4a4404c060 con 0x7f4a5406c7e0 2024-05-15T12:43:15.467 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.476+0000 7f4a50ff9640 1 --2- 172.21.15.117:0/3996711144 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f4a4006ceb0 0x7f4a4006f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:15.467 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.476+0000 7f4a537fe640 1 --2- 172.21.15.117:0/3996711144 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f4a4006ceb0 0x7f4a4006f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:15.467 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.477+0000 7f4a537fe640 10 monclient: get_auth_request con 0x7f4a4006ceb0 auth_method 0 2024-05-15T12:43:15.467 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.477+0000 7f4a50ff9640 10 client.4377.objecter ms_handle_connect 0x7f4a4006ceb0 2024-05-15T12:43:15.467 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.477+0000 7f4a537fe640 1 --2- 172.21.15.117:0/3996711144 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f4a4006ceb0 0x7f4a4006f370 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4a5414dc00 tx=0x7f4a48009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:15.480 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.490+0000 7f4a50ff9640 1 -- 172.21.15.117:0/3996711144 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f4a44078e20 con 0x7f4a5406c7e0 2024-05-15T12:43:15.480 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.490+0000 7f4a50ff9640 10 client.4377.objecter ms_dispatch 0x7f4a54000b90 osd_map(26..26 src has 1..26) v4 2024-05-15T12:43:15.480 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.490+0000 7f4a50ff9640 3 client.4377.objecter handle_osd_map got epochs [26,26] > 0 2024-05-15T12:43:15.481 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.490+0000 7f4a50ff9640 3 client.4377.objecter handle_osd_map decoding full epoch 26 2024-05-15T12:43:15.481 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.490+0000 7f4a50ff9640 20 client.4377.objecter dump_active .. 0 homeless 2024-05-15T12:43:15.481 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.490+0000 7f4a50ff9640 1 -- 172.21.15.117:0/3996711144 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f4a44040070 con 0x7f4a5406c7e0 2024-05-15T12:43:15.481 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.490+0000 7f4a50ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:15.481 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.490+0000 7f4a50ff9640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:43:15.651 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.661+0000 7f4a5a3fe640 1 -- 172.21.15.117:0/3996711144 --> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4a5414dc30 con 0x7f4a4006ceb0 2024-05-15T12:43:15.652 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.662+0000 7f4a50ff9640 1 -- 172.21.15.117:0/3996711144 <== mgr.4107 v2:172.21.15.117:6800/1785766523 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+35506 (secure 0 0 0) 0x7f4a5414dc30 con 0x7f4a4006ceb0 2024-05-15T12:43:15.653 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:43:15.653 INFO:teuthology.orchestra.run.smithi117.stderr:dumped all 2024-05-15T12:43:15.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.664+0000 7f4a5a3fe640 10 librados: watch_flush enter 2024-05-15T12:43:15.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.664+0000 7f4a5a3fe640 10 librados: watch_flush exit 2024-05-15T12:43:15.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.664+0000 7f4a5a3fe640 20 client.4377.objecter shutdown clearing up homeless session... 2024-05-15T12:43:15.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.664+0000 7f4a5a3fe640 10 client.4377.objecter successfully canceled tick 2024-05-15T12:43:15.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.664+0000 7f4a5a3fe640 1 -- 172.21.15.117:0/3996711144 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f4a4006ceb0 msgr2=0x7f4a4006f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:15.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.664+0000 7f4a5a3fe640 1 --2- 172.21.15.117:0/3996711144 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f4a4006ceb0 0x7f4a4006f370 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4a5414dc00 tx=0x7f4a48009040 comp rx=0 tx=0).stop 2024-05-15T12:43:15.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.664+0000 7f4a5a3fe640 10 monclient: shutdown 2024-05-15T12:43:15.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.664+0000 7f4a5a3fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:15.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.664+0000 7f4a5a3fe640 1 -- 172.21.15.117:0/3996711144 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4a5406c7e0 msgr2=0x7f4a541446b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:15.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.664+0000 7f4a5a3fe640 1 --2- 172.21.15.117:0/3996711144 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4a5406c7e0 0x7f4a541446b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4a440097d0 tx=0x7f4a440046c0 comp rx=0 tx=0).stop 2024-05-15T12:43:15.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.664+0000 7f4a5a3fe640 1 -- 172.21.15.117:0/3996711144 shutdown_connections 2024-05-15T12:43:15.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.664+0000 7f4a5a3fe640 1 --2- 172.21.15.117:0/3996711144 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4a54142bf0 0x7f4a541401b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:15.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.664+0000 7f4a5a3fe640 1 --2- 172.21.15.117:0/3996711144 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f4a4006ceb0 0x7f4a4006f370 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:15.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.664+0000 7f4a5a3fe640 1 --2- 172.21.15.117:0/3996711144 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4a5406cbc0 0x7f4a54144120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:15.656 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.664+0000 7f4a5a3fe640 1 --2- 172.21.15.117:0/3996711144 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4a5406c7e0 0x7f4a541446b0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:15.656 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.664+0000 7f4a5a3fe640 1 -- 172.21.15.117:0/3996711144 >> 172.21.15.117:0/3996711144 conn(0x7f4a540b5f40 msgr2=0x7f4a540b1710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:15.656 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.664+0000 7f4a5a3fe640 1 -- 172.21.15.117:0/3996711144 shutdown_connections 2024-05-15T12:43:15.656 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.664+0000 7f4a5a3fe640 1 -- 172.21.15.117:0/3996711144 wait complete. 2024-05-15T12:43:15.656 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.664+0000 7f4a5a3fe640 1 librados: shutdown 2024-05-15T12:43:15.665 INFO:teuthology.orchestra.run.smithi117.stdout:{"pg_ready":true,"pg_map":{"version":42,"stamp":"2024-05-15T12:43:13.910512+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218832,"kb_used_data":2664,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755888,"statfs":{"total":773094113280,"available":772870029312,"internally_reserved":0,"allocated":2727936,"data_stored":1468512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001128"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-15T12:43:10.884536+0000","last_change":"2024-05-15T12:43:09.429828+0000","last_active":"2024-05-15T12:43:10.884536+0000","last_peered":"2024-05-15T12:43:10.884536+0000","last_clean":"2024-05-15T12:43:10.884536+0000","last_became_active":"2024-05-15T12:43:09.424460+0000","last_became_peered":"2024-05-15T12:43:09.424460+0000","last_unstale":"2024-05-15T12:43:10.884536+0000","last_undegraded":"2024-05-15T12:43:10.884536+0000","last_fullsized":"2024-05-15T12:43:10.884536+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_clean_scrub_stamp":"2024-05-15T12:43:07.133352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:12:21.424590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-15T12:43:10.812177+0000","last_change":"2024-05-15T12:43:09.478046+0000","last_active":"2024-05-15T12:43:10.812177+0000","last_peered":"2024-05-15T12:43:10.812177+0000","last_clean":"2024-05-15T12:43:10.812177+0000","last_became_active":"2024-05-15T12:43:09.448237+0000","last_became_peered":"2024-05-15T12:43:09.448237+0000","last_unstale":"2024-05-15T12:43:10.812177+0000","last_undegraded":"2024-05-15T12:43:10.812177+0000","last_fullsized":"2024-05-15T12:43:10.812177+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_clean_scrub_stamp":"2024-05-15T12:43:07.133352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:11:17.055288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-15T12:43:10.878936+0000","last_change":"2024-05-15T12:43:08.778833+0000","last_active":"2024-05-15T12:43:10.878936+0000","last_peered":"2024-05-15T12:43:10.878936+0000","last_clean":"2024-05-15T12:43:10.878936+0000","last_became_active":"2024-05-15T12:43:08.773614+0000","last_became_peered":"2024-05-15T12:43:08.773614+0000","last_unstale":"2024-05-15T12:43:10.878936+0000","last_undegraded":"2024-05-15T12:43:10.878936+0000","last_fullsized":"2024-05-15T12:43:10.878936+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_clean_scrub_stamp":"2024-05-15T12:43:07.133352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:22:59.785687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-15T12:43:10.806149+0000","last_change":"2024-05-15T12:43:09.384765+0000","last_active":"2024-05-15T12:43:10.806149+0000","last_peered":"2024-05-15T12:43:10.806149+0000","last_clean":"2024-05-15T12:43:10.806149+0000","last_became_active":"2024-05-15T12:43:09.377725+0000","last_became_peered":"2024-05-15T12:43:09.377725+0000","last_unstale":"2024-05-15T12:43:10.806149+0000","last_undegraded":"2024-05-15T12:43:10.806149+0000","last_fullsized":"2024-05-15T12:43:10.806149+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_clean_scrub_stamp":"2024-05-15T12:43:07.133352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:57:59.671602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-15T12:43:10.861367+0000","last_change":"2024-05-15T12:43:09.375465+0000","last_active":"2024-05-15T12:43:10.861367+0000","last_peered":"2024-05-15T12:43:10.861367+0000","last_clean":"2024-05-15T12:43:10.861367+0000","last_became_active":"2024-05-15T12:43:09.363876+0000","last_became_peered":"2024-05-15T12:43:09.363876+0000","last_unstale":"2024-05-15T12:43:10.861367+0000","last_undegraded":"2024-05-15T12:43:10.861367+0000","last_fullsized":"2024-05-15T12:43:10.861367+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_clean_scrub_stamp":"2024-05-15T12:43:07.133352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:27:46.599385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-05-15T12:43:10.054499+0000","last_change":"2024-05-15T12:43:08.634999+0000","last_active":"2024-05-15T12:43:10.054499+0000","last_peered":"2024-05-15T12:43:10.054499+0000","last_clean":"2024-05-15T12:43:10.054499+0000","last_became_active":"2024-05-15T12:43:08.600745+0000","last_became_peered":"2024-05-15T12:43:08.600745+0000","last_unstale":"2024-05-15T12:43:10.054499+0000","last_undegraded":"2024-05-15T12:43:10.054499+0000","last_fullsized":"2024-05-15T12:43:10.054499+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_clean_scrub_stamp":"2024-05-15T12:43:07.133352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T15:01:15.586689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-15T12:43:10.837850+0000","last_change":"2024-05-15T12:43:08.704425+0000","last_active":"2024-05-15T12:43:10.837850+0000","last_peered":"2024-05-15T12:43:10.837850+0000","last_clean":"2024-05-15T12:43:10.837850+0000","last_became_active":"2024-05-15T12:43:08.677751+0000","last_became_peered":"2024-05-15T12:43:08.677751+0000","last_unstale":"2024-05-15T12:43:10.837850+0000","last_undegraded":"2024-05-15T12:43:10.837850+0000","last_fullsized":"2024-05-15T12:43:10.837850+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_clean_scrub_stamp":"2024-05-15T12:43:07.133352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-17T00:22:51.843061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-15T12:43:10.873093+0000","last_change":"2024-05-15T12:43:09.453018+0000","last_active":"2024-05-15T12:43:10.873093+0000","last_peered":"2024-05-15T12:43:10.873093+0000","last_clean":"2024-05-15T12:43:10.873093+0000","last_became_active":"2024-05-15T12:43:09.445878+0000","last_became_peered":"2024-05-15T12:43:09.445878+0000","last_unstale":"2024-05-15T12:43:10.873093+0000","last_undegraded":"2024-05-15T12:43:10.873093+0000","last_fullsized":"2024-05-15T12:43:10.873093+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_clean_scrub_stamp":"2024-05-15T12:43:07.133352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:55:05.766298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'76","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-15T12:43:11.944076+0000","last_change":"2024-05-15T12:43:04.763767+0000","last_active":"2024-05-15T12:43:11.944076+0000","last_peered":"2024-05-15T12:43:11.944076+0000","last_clean":"2024-05-15T12:43:11.944076+0000","last_became_active":"2024-05-15T12:43:04.148735+0000","last_became_peered":"2024-05-15T12:43:04.148735+0000","last_unstale":"2024-05-15T12:43:11.944076+0000","last_undegraded":"2024-05-15T12:43:11.944076+0000","last_fullsized":"2024-05-15T12:43:11.944076+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:42:53.071197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:42:53.071197+0000","last_clean_scrub_stamp":"2024-05-15T12:42:53.071197+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:45:23.691461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":20,"seq":85899345930,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27656,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344184,"statfs":{"total":96636764160,"available":96608444416,"internally_reserved":0,"allocated":651264,"data_stored":486255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":20,"seq":85899345932,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":237568,"data_stored":82667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":20,"seq":85899345931,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":237568,"data_stored":82667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":19,"seq":81604378632,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":237568,"data_stored":82667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":19,"seq":81604378634,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":237568,"data_stored":82667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509448,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":249856,"data_stored":88415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542151,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27644,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344196,"statfs":{"total":96636764160,"available":96608456704,"internally_reserved":0,"allocated":638976,"data_stored":480507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":237568,"data_stored":82667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T12:43:15.666 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-05-15T12:43:15.666 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-05-15T12:43:15.666 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-05-15T12:43:15.667 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-05-15T12:43:15.764 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.774+0000 7f5064d17640 10 monclient: get_monmap_and_config 2024-05-15T12:43:15.765 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.774+0000 7f5064d17640 10 monclient: build_initial_monmap 2024-05-15T12:43:15.765 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.774+0000 7f5064d17640 10 monclient: monmap: 2024-05-15T12:43:15.765 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:15.765 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:15.765 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:43:15.775666+0000 2024-05-15T12:43:15.765 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:43:15.775666+0000 2024-05-15T12:43:15.765 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:15.765 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:15.765 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T12:43:15.765 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T12:43:15.765 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T12:43:15.765 INFO:teuthology.orchestra.run.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T12:43:15.765 INFO:teuthology.orchestra.run.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T12:43:15.765 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:15.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.775+0000 7f5064d17640 1 Processor -- start 2024-05-15T12:43:15.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.775+0000 7f5064d17640 1 -- start start 2024-05-15T12:43:15.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.775+0000 7f5064d17640 10 monclient: init 2024-05-15T12:43:15.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.775+0000 7f5064d17640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:15.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.775+0000 7f5064d17640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-05-15T12:43:15.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.775+0000 7f5064d17640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f506006c900 0x7f50600aeab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:15.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.775+0000 7f5064d17640 10 monclient(hunting): picked mon.noname-b con 0x7f506006c900 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:43:15.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.776+0000 7f5064d17640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f50600aeff0 addr v1:172.21.15.117:6789/0 2024-05-15T12:43:15.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.776+0000 7f5064d17640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f50600af410 0x7f5060146590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:15.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.776+0000 7f5064d17640 10 monclient(hunting): picked mon.noname-a con 0x7f50600af410 addr v2:172.21.15.117:3300/0 2024-05-15T12:43:15.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.776+0000 7f5064d17640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50600ae0d0 con 0x7f50600aeff0 2024-05-15T12:43:15.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.776+0000 7f5064d17640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:15.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.776+0000 7f5064d17640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f506006cce0 con 0x7f50600af410 2024-05-15T12:43:15.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.776+0000 7f5064d17640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:15.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.776+0000 7f5064d17640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50600b0500 con 0x7f506006c900 2024-05-15T12:43:15.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.776+0000 7f5064d17640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:15.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.776+0000 7f5064d17640 10 monclient(hunting): authenticate will time out at 1359.017834s 2024-05-15T12:43:15.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.776+0000 7f505ed9d640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f50600af410 0x7f5060146590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:15.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.776+0000 7f505e59c640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f506006c900 0x7f50600aeab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:15.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.777+0000 7f505e59c640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f506006c900 0x7f50600aeab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.117:49200/0 (socket says 172.21.15.117:49200) 2024-05-15T12:43:15.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.777+0000 7f505e59c640 1 -- 172.21.15.117:0/2244376193 learned_addr learned my addr 172.21.15.117:0/2244376193 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:43:15.770 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.777+0000 7f505e59c640 10 monclient(hunting): get_auth_request con 0x7f506006c900 auth_method 0 2024-05-15T12:43:15.770 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.777+0000 7f505e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:15.770 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.777+0000 7f505e59c640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:15.770 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.777+0000 7f505e59c640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:15.770 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.777+0000 7f505ed9d640 10 monclient(hunting): get_auth_request con 0x7f50600af410 auth_method 0 2024-05-15T12:43:15.770 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.777+0000 7f505ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:15.770 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.777+0000 7f505ed9d640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:15.770 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.777+0000 7f505ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:15.770 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.777+0000 7f505e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:15.770 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.777+0000 7f505e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:15.770 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.777+0000 7f505e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:15.770 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.778+0000 7f505ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:15.770 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.778+0000 7f505ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:15.770 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.778+0000 7f505ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:15.771 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.779+0000 7f505e59c640 10 monclient(hunting): handle_auth_done global_id 4328 payload 274 2024-05-15T12:43:15.771 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.779+0000 7f505e59c640 1 -- 172.21.15.117:0/2244376193 >> v2:172.21.15.117:3300/0 conn(0x7f50600af410 msgr2=0x7f5060146590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:43:15.771 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.779+0000 7f505e59c640 1 --2- 172.21.15.117:0/2244376193 >> v2:172.21.15.117:3300/0 conn(0x7f50600af410 0x7f5060146590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:15.772 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.779+0000 7f505e59c640 1 -- 172.21.15.117:0/2244376193 >> v1:172.21.15.117:6789/0 conn(0x7f50600aeff0 legacy=0x7f5060145e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:15.772 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.779+0000 7f505e59c640 10 monclient: _finish_hunting 0 2024-05-15T12:43:15.772 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.779+0000 7f505e59c640 1 monclient: found mon.noname-b 2024-05-15T12:43:15.772 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.779+0000 7f505e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:15.772 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.779+0000 7f505e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.117:3301/0 2024-05-15T12:43:15.772 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.779+0000 7f505e59c640 1 -- 172.21.15.117:0/2244376193 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50600ae3b0 con 0x7f506006c900 2024-05-15T12:43:15.772 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.781+0000 7f505e59c640 1 --2- 172.21.15.117:0/2244376193 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f506006c900 0x7f50600aeab0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5050009870 tx=0x7f505002b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe7359969435cd6 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:15.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.784+0000 7f505dd9b640 1 -- 172.21.15.117:0/2244376193 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50500317c0 con 0x7f506006c900 2024-05-15T12:43:15.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.784+0000 7f505dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:15.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.784+0000 7f505dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:43:15.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.784+0000 7f505dd9b640 10 monclient: dump: 2024-05-15T12:43:15.775 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:15.775 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:15.775 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:15.775 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:15.775 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:15.775 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:15.775 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:15.775 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:15.775 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:15.776 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:15.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.784+0000 7f505dd9b640 10 monclient: _finish_auth 0 2024-05-15T12:43:15.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.784+0000 7f505dd9b640 10 monclient: _check_auth_tickets 2024-05-15T12:43:15.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.784+0000 7f505dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:15.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.784+0000 7f505dd9b640 1 -- 172.21.15.117:0/2244376193 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5050037070 con 0x7f506006c900 2024-05-15T12:43:15.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.784+0000 7f505dd9b640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:15.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.784+0000 7f505dd9b640 1 -- 172.21.15.117:0/2244376193 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5050031b80 con 0x7f506006c900 2024-05-15T12:43:15.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.784+0000 7f5064d17640 5 monclient: authenticate success, global_id 4328 2024-05-15T12:43:15.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.784+0000 7f5064d17640 10 monclient: get_monmap_and_config success 2024-05-15T12:43:15.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.784+0000 7f5064d17640 4 set_mon_vals no callback set 2024-05-15T12:43:15.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.784+0000 7f505dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:15.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.785+0000 7f505dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T12:43:15.777 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.785+0000 7f505dd9b640 10 monclient: dump: 2024-05-15T12:43:15.777 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:15.777 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:15.777 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:15.777 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:15.777 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:15.777 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:15.777 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:15.777 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:15.777 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:15.777 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:15.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.785+0000 7f5064d17640 10 monclient: shutdown 2024-05-15T12:43:15.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.785+0000 7f5064d17640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:15.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.785+0000 7f5064d17640 1 -- 172.21.15.117:0/2244376193 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f506006c900 msgr2=0x7f50600aeab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:15.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.785+0000 7f5064d17640 1 --2- 172.21.15.117:0/2244376193 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f506006c900 0x7f50600aeab0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5050009870 tx=0x7f505002b860 comp rx=0 tx=0).stop 2024-05-15T12:43:15.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.785+0000 7f5064d17640 1 -- 172.21.15.117:0/2244376193 shutdown_connections 2024-05-15T12:43:15.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.785+0000 7f5064d17640 1 --2- 172.21.15.117:0/2244376193 >> v2:172.21.15.117:3300/0 conn(0x7f50600af410 0x7f5060146590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:15.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.785+0000 7f5064d17640 1 --2- 172.21.15.117:0/2244376193 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f506006c900 0x7f50600aeab0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:15.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.785+0000 7f5064d17640 1 -- 172.21.15.117:0/2244376193 >> 172.21.15.117:0/2244376193 conn(0x7f50600b2ac0 msgr2=0x7f50600b2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:15.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.785+0000 7f5064d17640 1 -- 172.21.15.117:0/2244376193 shutdown_connections 2024-05-15T12:43:15.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.785+0000 7f5064d17640 1 -- 172.21.15.117:0/2244376193 wait complete. 2024-05-15T12:43:15.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.786+0000 7f5064d17640 10 monclient: build_initial_monmap 2024-05-15T12:43:15.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.786+0000 7f5064d17640 10 monclient: monmap: 2024-05-15T12:43:15.782 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T12:43:15.782 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:43:15.782 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-05-15T12:43:15.782 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-05-15T12:43:15.782 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T12:43:15.782 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:15.782 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T12:43:15.782 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T12:43:15.782 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T12:43:15.782 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:15.782 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.786+0000 7f5064d17640 1 librados: starting msgr at 2024-05-15T12:43:15.782 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.786+0000 7f5064d17640 1 librados: starting objecter 2024-05-15T12:43:15.782 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.787+0000 7f5064d17640 1 Processor -- start 2024-05-15T12:43:15.782 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.787+0000 7f5064d17640 1 -- start start 2024-05-15T12:43:15.782 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.787+0000 7f5064d17640 1 librados: setting wanted keys 2024-05-15T12:43:15.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.787+0000 7f5064d17640 1 librados: calling monclient init 2024-05-15T12:43:15.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.787+0000 7f5064d17640 10 monclient: init 2024-05-15T12:43:15.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.787+0000 7f5064d17640 10 monclient: _reopen_session rank -1 2024-05-15T12:43:15.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.787+0000 7f5064d17640 10 monclient: _add_conns ranks=[2,0,1] 2024-05-15T12:43:15.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.787+0000 7f5064d17640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f506006c900 0x7f50601db070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:15.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.787+0000 7f5064d17640 10 monclient(hunting): picked mon.noname-c con 0x7f506006c900 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:43:15.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.787+0000 7f5064d17640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f50600aeff0 0x7f506013ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:15.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.787+0000 7f5064d17640 10 monclient(hunting): picked mon.noname-a con 0x7f50600aeff0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T12:43:15.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.787+0000 7f5064d17640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f50600af410 0x7f506013f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:15.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.787+0000 7f5064d17640 10 monclient(hunting): picked mon.noname-b con 0x7f50600af410 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:43:15.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.787+0000 7f5064d17640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:15.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.787+0000 7f5064d17640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5060146ad0 con 0x7f50600aeff0 2024-05-15T12:43:15.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.787+0000 7f5064d17640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:15.784 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.787+0000 7f5064d17640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f506006d730 con 0x7f50600af410 2024-05-15T12:43:15.784 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.787+0000 7f5064d17640 10 monclient(hunting): start opening mon connection 2024-05-15T12:43:15.784 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.787+0000 7f5064d17640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5060146c50 con 0x7f506006c900 2024-05-15T12:43:15.784 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.787+0000 7f5064d17640 10 monclient(hunting): _renew_subs 2024-05-15T12:43:15.784 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.787+0000 7f5064d17640 10 monclient(hunting): authenticate will time out at 1359.029516s 2024-05-15T12:43:15.784 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.788+0000 7f505e59c640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f506006c900 0x7f50601db070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:15.784 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.788+0000 7f5057fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f50600aeff0 0x7f506013ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:15.784 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.788+0000 7f505ed9d640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f50600af410 0x7f506013f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:15.784 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.788+0000 7f505e59c640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f506006c900 0x7f50601db070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.117:49216/0 (socket says 172.21.15.117:49216) 2024-05-15T12:43:15.784 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.788+0000 7f5057fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f50600aeff0 0x7f506013ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:60442/0 (socket says 172.21.15.117:60442) 2024-05-15T12:43:15.784 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.788+0000 7f505e59c640 1 -- 172.21.15.117:0/143329437 learned_addr learned my addr 172.21.15.117:0/143329437 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:43:15.784 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.788+0000 7f505e59c640 10 monclient(hunting): get_auth_request con 0x7f506006c900 auth_method 0 2024-05-15T12:43:15.784 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.788+0000 7f505e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:15.784 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.788+0000 7f505e59c640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:15.785 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.788+0000 7f505e59c640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:15.785 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.788+0000 7f5057fff640 10 monclient(hunting): get_auth_request con 0x7f50600aeff0 auth_method 0 2024-05-15T12:43:15.785 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.788+0000 7f5057fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:15.785 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.788+0000 7f5057fff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:15.785 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.788+0000 7f5057fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:15.785 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.788+0000 7f505ed9d640 10 monclient(hunting): get_auth_request con 0x7f50600af410 auth_method 0 2024-05-15T12:43:15.785 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.788+0000 7f505ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:43:15.785 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.788+0000 7f505ed9d640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:43:15.785 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.788+0000 7f505ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:43:15.785 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.789+0000 7f505e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:15.785 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.789+0000 7f505e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:15.785 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.789+0000 7f505e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:15.785 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.789+0000 7f5057fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:15.786 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.789+0000 7f5057fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:15.786 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.789+0000 7f5057fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:15.786 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.789+0000 7f505ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:43:15.786 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.789+0000 7f505ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:43:15.786 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.789+0000 7f505ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:43:15.787 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.792+0000 7f5057fff640 10 monclient(hunting): handle_auth_done global_id 4386 payload 824 2024-05-15T12:43:15.787 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.792+0000 7f5057fff640 1 -- 172.21.15.117:0/143329437 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f506006c900 msgr2=0x7f50601db070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:15.787 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.792+0000 7f5057fff640 1 --2- 172.21.15.117:0/143329437 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f506006c900 0x7f50601db070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:15.787 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.792+0000 7f5057fff640 1 -- 172.21.15.117:0/143329437 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f50600af410 msgr2=0x7f506013f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:15.787 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.792+0000 7f5057fff640 1 --2- 172.21.15.117:0/143329437 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f50600af410 0x7f506013f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:15.787 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.792+0000 7f5057fff640 10 monclient: _finish_hunting 0 2024-05-15T12:43:15.787 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.792+0000 7f5057fff640 1 monclient: found mon.noname-a 2024-05-15T12:43:15.787 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.792+0000 7f5057fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:43:15.788 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.792+0000 7f5057fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.117:3300/0 2024-05-15T12:43:15.788 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.792+0000 7f5057fff640 1 -- 172.21.15.117:0/143329437 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5060121f60 con 0x7f50600aeff0 2024-05-15T12:43:15.788 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.792+0000 7f505e59c640 1 --2- 172.21.15.117:0/143329437 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f506006c900 0x7f50601db070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-15T12:43:15.788 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.794+0000 7f5057fff640 1 --2- 172.21.15.117:0/143329437 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f50600aeff0 0x7f506013ef50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f504800c340 tx=0x7f504800c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:15.788 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.794+0000 7f5056ffd640 10 client.?.objecter ms_handle_connect 0x7f50600aeff0 2024-05-15T12:43:15.788 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.794+0000 7f5056ffd640 10 client.?.objecter resend_mon_ops 2024-05-15T12:43:15.789 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.797+0000 7f5056ffd640 1 -- 172.21.15.117:0/143329437 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50480052d0 con 0x7f50600aeff0 2024-05-15T12:43:15.789 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.797+0000 7f5056ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:15.789 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.797+0000 7f5056ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T12:43:15.789 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.797+0000 7f5056ffd640 10 monclient: dump: 2024-05-15T12:43:15.790 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:15.790 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:15.790 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:15.790 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:15.790 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:15.790 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:15.790 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:15.790 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:15.790 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:15.790 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:15.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.797+0000 7f5056ffd640 10 monclient: _finish_auth 0 2024-05-15T12:43:15.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.797+0000 7f5056ffd640 10 monclient: _check_auth_tickets 2024-05-15T12:43:15.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.797+0000 7f5056ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:43:15.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.797+0000 7f5056ffd640 1 -- 172.21.15.117:0/143329437 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5048005490 con 0x7f50600aeff0 2024-05-15T12:43:15.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.797+0000 7f5064d17640 5 monclient: authenticate success, global_id 4386 2024-05-15T12:43:15.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.797+0000 7f5064d17640 10 monclient: _renew_subs 2024-05-15T12:43:15.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.797+0000 7f5064d17640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:43:15.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.797+0000 7f5064d17640 1 -- 172.21.15.117:0/143329437 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50601221d0 con 0x7f50600aeff0 2024-05-15T12:43:15.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.797+0000 7f5056ffd640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:43:15.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.797+0000 7f5056ffd640 1 -- 172.21.15.117:0/143329437 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5048007260 con 0x7f50600aeff0 2024-05-15T12:43:15.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.797+0000 7f5056ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:43:15.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.797+0000 7f5056ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T12:43:15.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.797+0000 7f5056ffd640 10 monclient: dump: 2024-05-15T12:43:15.791 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T12:43:15.791 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:43:15.791 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:15.791 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:43:15.791 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T12:43:15.792 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T12:43:15.792 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:43:15.792 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:43:15.792 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:43:15.792 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T12:43:15.792 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.797+0000 7f505d59a640 4 set_mon_vals no callback set 2024-05-15T12:43:15.792 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.798+0000 7f5064d17640 10 client.4386.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:43:15.792 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.798+0000 7f5064d17640 10 monclient: _renew_subs 2024-05-15T12:43:15.792 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.798+0000 7f5064d17640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:43:15.792 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.798+0000 7f5064d17640 1 -- 172.21.15.117:0/143329437 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f506013f9d0 con 0x7f50600aeff0 2024-05-15T12:43:15.792 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.798+0000 7f5064d17640 1 librados: init done 2024-05-15T12:43:15.792 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.798+0000 7f5064d17640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:15.792 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.798+0000 7f5064d17640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:15.792 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.798+0000 7f5064d17640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:43:15.793 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.798+0000 7f5064d17640 1 -- 172.21.15.117:0/143329437 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f506004f0e0 con 0x7f50600aeff0 2024-05-15T12:43:15.808 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.818+0000 7f5056ffd640 1 -- 172.21.15.117:0/143329437 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f5048007400 con 0x7f50600aeff0 2024-05-15T12:43:15.809 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.818+0000 7f5056ffd640 1 --2- 172.21.15.117:0/143329437 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f504006ceb0 0x7f504006f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:43:15.809 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.818+0000 7f505e59c640 1 --2- 172.21.15.117:0/143329437 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f504006ceb0 0x7f504006f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:43:15.809 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.818+0000 7f505e59c640 10 monclient: get_auth_request con 0x7f504006ceb0 auth_method 0 2024-05-15T12:43:15.809 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.819+0000 7f5056ffd640 10 client.4386.objecter ms_handle_connect 0x7f504006ceb0 2024-05-15T12:43:15.809 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:15.819+0000 7f505e59c640 1 --2- 172.21.15.117:0/143329437 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f504006ceb0 0x7f504006f370 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f5050034040 tx=0x7f505003d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:43:16.050 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:16.060+0000 7f5056ffd640 1 -- 172.21.15.117:0/143329437 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f5048056360 con 0x7f50600aeff0 2024-05-15T12:43:16.050 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:16.060+0000 7f5056ffd640 10 client.4386.objecter ms_dispatch 0x7f5060000b90 osd_map(26..26 src has 1..26) v4 2024-05-15T12:43:16.050 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:16.060+0000 7f5056ffd640 3 client.4386.objecter handle_osd_map got epochs [26,26] > 0 2024-05-15T12:43:16.051 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:16.060+0000 7f5056ffd640 3 client.4386.objecter handle_osd_map decoding full epoch 26 2024-05-15T12:43:16.051 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:16.060+0000 7f5056ffd640 20 client.4386.objecter dump_active .. 0 homeless 2024-05-15T12:43:16.051 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:16.060+0000 7f5056ffd640 1 -- 172.21.15.117:0/143329437 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f50480582c0 con 0x7f50600aeff0 2024-05-15T12:43:16.051 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:16.060+0000 7f5056ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:43:16.051 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:16.060+0000 7f5056ffd640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:43:16.301 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:16.311+0000 7f5064d17640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-05-15T12:43:16.301 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:16.311+0000 7f5064d17640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-05-15T12:43:16.301 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:16.311+0000 7f5064d17640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T12:43:16.301 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:16.311+0000 7f5064d17640 1 -- 172.21.15.117:0/143329437 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5060120060 con 0x7f50600aeff0 2024-05-15T12:43:16.309 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:16.318+0000 7f5056ffd640 1 -- 172.21.15.117:0/143329437 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f504805c360 con 0x7f50600aeff0 2024-05-15T12:43:16.309 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:16.318+0000 7f5056ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-05-15T12:43:16.309 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:16.318+0000 7f5056ffd640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:43:16.310 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T12:43:16.310 INFO:teuthology.orchestra.run.smithi117.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-05-15T12:43:16.312 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:16.321+0000 7f5064d17640 10 librados: watch_flush enter 2024-05-15T12:43:16.312 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:16.321+0000 7f5064d17640 10 librados: watch_flush exit 2024-05-15T12:43:16.312 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:16.321+0000 7f5064d17640 20 client.4386.objecter shutdown clearing up homeless session... 2024-05-15T12:43:16.312 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:16.321+0000 7f5064d17640 10 client.4386.objecter successfully canceled tick 2024-05-15T12:43:16.312 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:16.321+0000 7f5064d17640 1 -- 172.21.15.117:0/143329437 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f504006ceb0 msgr2=0x7f504006f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:16.312 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:16.321+0000 7f5064d17640 1 --2- 172.21.15.117:0/143329437 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f504006ceb0 0x7f504006f370 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f5050034040 tx=0x7f505003d040 comp rx=0 tx=0).stop 2024-05-15T12:43:16.312 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:16.321+0000 7f5064d17640 10 monclient: shutdown 2024-05-15T12:43:16.312 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:16.321+0000 7f5064d17640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:43:16.313 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:16.321+0000 7f5064d17640 1 -- 172.21.15.117:0/143329437 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f50600aeff0 msgr2=0x7f506013ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:43:16.313 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:16.321+0000 7f5064d17640 1 --2- 172.21.15.117:0/143329437 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f50600aeff0 0x7f506013ef50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f504800c340 tx=0x7f504800c800 comp rx=0 tx=0).stop 2024-05-15T12:43:16.313 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:16.321+0000 7f5064d17640 1 -- 172.21.15.117:0/143329437 shutdown_connections 2024-05-15T12:43:16.313 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:16.321+0000 7f5064d17640 1 --2- 172.21.15.117:0/143329437 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f504006ceb0 0x7f504006f370 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:16.313 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:16.321+0000 7f5064d17640 1 --2- 172.21.15.117:0/143329437 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f50600af410 0x7f506013f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:16.313 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:16.321+0000 7f5064d17640 1 --2- 172.21.15.117:0/143329437 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f50600aeff0 0x7f506013ef50 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:16.313 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:16.321+0000 7f5064d17640 1 --2- 172.21.15.117:0/143329437 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f506006c900 0x7f50601db070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:43:16.313 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:16.321+0000 7f5064d17640 1 -- 172.21.15.117:0/143329437 >> 172.21.15.117:0/143329437 conn(0x7f50600b2ac0 msgr2=0x7f50600b2ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:43:16.313 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:16.321+0000 7f5064d17640 1 -- 172.21.15.117:0/143329437 shutdown_connections 2024-05-15T12:43:16.313 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:16.321+0000 7f5064d17640 1 -- 172.21.15.117:0/143329437 wait complete. 2024-05-15T12:43:16.313 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T12:43:16.321+0000 7f5064d17640 1 librados: shutdown 2024-05-15T12:43:16.323 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-05-15T12:43:16.324 INFO:teuthology.run_tasks:Running task workunit... 2024-05-15T12:43:16.337 INFO:tasks.workunit:Pulling workunits from ref ae2481253f9b2fa3451954e615eedb67ebdd5314 2024-05-15T12:43:16.337 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-05-15T12:43:16.337 DEBUG:teuthology.orchestra.run.smithi117:> stat -- /home/ubuntu/cephtest/mnt.0 2024-05-15T12:43:16.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T12:43:16.352 INFO:teuthology.orchestra.run.smithi117.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-05-15T12:43:16.352 DEBUG:teuthology.orchestra.run.smithi117:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-05-15T12:43:16.408 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-05-15T12:43:16.408 DEBUG:teuthology.orchestra.run.smithi117:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-05-15T12:43:16.464 INFO:tasks.workunit:timeout=6h 2024-05-15T12:43:16.464 INFO:tasks.workunit:cleanup=True 2024-05-15T12:43:16.465 DEBUG:teuthology.orchestra.run.smithi117:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout ae2481253f9b2fa3451954e615eedb67ebdd5314 2024-05-15T12:43:16.520 INFO:tasks.workunit.client.0.smithi117.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-05-15T12:43:22.209 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:01:29.388 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:43:22.211 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:01:29.390 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:43:22.217 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:01:29.389 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:43:22.220 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:01:29.392 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:43:22.258 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:01:29.437 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:43:22.267 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:01:29.446 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:43:22.267 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:01:29.439 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:43:22.269 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:01:29.448 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:43:22.316 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:01:29.495 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:43:42.245 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:01:49.423 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:43:42.247 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:01:49.426 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:43:42.261 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:01:49.439 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:43:42.297 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:01:49.468 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:43:42.297 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:01:49.476 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:43:42.299 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:01:49.478 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:43:42.300 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:01:49.472 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:43:42.313 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:01:49.485 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:43:42.314 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:01:49.492 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:02.288 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:02:09.466 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:02.292 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:02:09.471 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:02.311 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:02:09.490 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:02.343 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:02:09.514 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:02.344 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:02:09.516 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:02.350 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:02:09.529 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:02.351 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:02:09.530 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:02.357 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:02:09.529 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:02.369 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:02:09.548 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:22.401 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:02:29.572 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:22.402 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:02:29.573 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:22.415 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:02:29.586 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:22.427 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:02:29.606 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:22.432 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:02:29.611 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:22.453 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:02:29.632 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:22.511 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:02:29.689 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:22.516 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:02:29.695 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:22.535 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:02:29.714 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:42.479 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:02:49.657 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:42.483 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:02:49.662 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:42.485 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:02:49.664 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:42.498 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:02:49.677 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:42.534 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:02:49.705 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:42.538 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:02:49.715 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:42.538 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:02:49.708 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:42.548 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:02:49.720 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:42.555 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:02:49.734 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:44.243 INFO:tasks.workunit.client.0.smithi117.stderr:Updating files: 98% (12089/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-05-15T12:44:44.597 INFO:tasks.workunit.client.0.smithi117.stderr:Note: switching to 'ae2481253f9b2fa3451954e615eedb67ebdd5314'. 2024-05-15T12:44:44.598 INFO:tasks.workunit.client.0.smithi117.stderr: 2024-05-15T12:44:44.598 INFO:tasks.workunit.client.0.smithi117.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-05-15T12:44:44.598 INFO:tasks.workunit.client.0.smithi117.stderr:changes and commit them, and you can discard any commits you make in this 2024-05-15T12:44:44.598 INFO:tasks.workunit.client.0.smithi117.stderr:state without impacting any branches by switching back to a branch. 2024-05-15T12:44:44.598 INFO:tasks.workunit.client.0.smithi117.stderr: 2024-05-15T12:44:44.598 INFO:tasks.workunit.client.0.smithi117.stderr:If you want to create a new branch to retain commits you create, you may 2024-05-15T12:44:44.598 INFO:tasks.workunit.client.0.smithi117.stderr:do so (now or later) by using -c with the switch command. Example: 2024-05-15T12:44:44.598 INFO:tasks.workunit.client.0.smithi117.stderr: 2024-05-15T12:44:44.598 INFO:tasks.workunit.client.0.smithi117.stderr: git switch -c 2024-05-15T12:44:44.598 INFO:tasks.workunit.client.0.smithi117.stderr: 2024-05-15T12:44:44.598 INFO:tasks.workunit.client.0.smithi117.stderr:Or undo this operation with: 2024-05-15T12:44:44.598 INFO:tasks.workunit.client.0.smithi117.stderr: 2024-05-15T12:44:44.598 INFO:tasks.workunit.client.0.smithi117.stderr: git switch - 2024-05-15T12:44:44.598 INFO:tasks.workunit.client.0.smithi117.stderr: 2024-05-15T12:44:44.599 INFO:tasks.workunit.client.0.smithi117.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-05-15T12:44:44.599 INFO:tasks.workunit.client.0.smithi117.stderr: 2024-05-15T12:44:44.599 INFO:tasks.workunit.client.0.smithi117.stderr:HEAD is now at ae2481253f9 DNM: teuto retrigger 2024-05-15T12:44:44.602 DEBUG:teuthology.orchestra.run.smithi117:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-05-15T12:44:44.702 INFO:tasks.workunit.client.0.smithi117.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-05-15T12:44:44.704 INFO:tasks.workunit.client.0.smithi117.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-05-15T12:44:44.704 INFO:tasks.workunit.client.0.smithi117.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-05-15T12:44:45.280 INFO:tasks.workunit.client.0.smithi117.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-05-15T12:44:45.356 INFO:tasks.workunit.client.0.smithi117.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-05-15T12:44:45.395 INFO:tasks.workunit.client.0.smithi117.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-05-15T12:44:45.397 INFO:tasks.workunit.client.0.smithi117.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-05-15T12:44:45.397 INFO:tasks.workunit.client.0.smithi117.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-05-15T12:44:45.438 INFO:tasks.workunit.client.0.smithi117.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-05-15T12:44:45.441 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-05-15T12:44:45.441 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-05-15T12:44:45.457 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-05-15T12:44:45.458 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-05-15T12:44:45.458 DEBUG:teuthology.orchestra.run.smithi117:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ae2481253f9b2fa3451954e615eedb67ebdd5314 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 ALLOW_TIMEOUTS=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-05-15T12:44:45.518 INFO:tasks.workunit.client.0.smithi117.stderr:+ parallel=1 2024-05-15T12:44:45.518 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' '' = --serial ']' 2024-05-15T12:44:45.518 INFO:tasks.workunit.client.0.smithi117.stderr:+ crimson=0 2024-05-15T12:44:45.518 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' '' = --crimson ']' 2024-05-15T12:44:45.518 INFO:tasks.workunit.client.0.smithi117.stderr:+ color= 2024-05-15T12:44:45.518 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -t 1 ']' 2024-05-15T12:44:45.519 INFO:tasks.workunit.client.0.smithi117.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-05-15T12:44:45.519 INFO:tasks.workunit.client.0.smithi117.stderr:+ GTEST_OUTPUT_DIR=/home/ubuntu/cephtest/archive/unit_test_xml_report 2024-05-15T12:44:45.519 INFO:tasks.workunit.client.0.smithi117.stderr:+ mkdir -p /home/ubuntu/cephtest/archive/unit_test_xml_report 2024-05-15T12:44:45.520 INFO:tasks.workunit.client.0.smithi117.stderr:+ declare -A pids 2024-05-15T12:44:45.520 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-15T12:44:45.520 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:44:45.520 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s api_aio 2024-05-15T12:44:45.520 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' api_aio' 2024-05-15T12:44:45.520 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo api_aio 2024-05-15T12:44:45.520 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-05-15T12:44:45.521 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=api_aio 2024-05-15T12:44:45.521 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=39858 2024-05-15T12:44:45.521 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test api_aio on pid 39858' 2024-05-15T12:44:45.521 INFO:tasks.workunit.client.0.smithi117.stdout:test api_aio on pid 39858 2024-05-15T12:44:45.524 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=39858 2024-05-15T12:44:45.524 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-15T12:44:45.524 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:44:45.525 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2024-05-15T12:44:45.525 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s api_aio_pp 2024-05-15T12:44:45.525 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' api_aio_pp' 2024-05-15T12:44:45.525 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo api_aio_pp 2024-05-15T12:44:45.525 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-05-15T12:44:45.525 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:44:45.525 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-05-15T12:44:45.525 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-05-15T12:44:45.525 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:44:45.525 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.525 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-05-15T12:44:45.525 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-05-15T12:44:45.525 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:44:45.525 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-05-15T12:44:45.526 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.526 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:44:45.526 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.526 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:44:45.526 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:44:45.526 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.526 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:44:45.526 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.526 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:44:45.526 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:45.526 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:45.526 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.526 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:44:45.526 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.526 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:44:45.527 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.527 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=api_aio_pp 2024-05-15T12:44:45.527 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=39865 2024-05-15T12:44:45.527 INFO:tasks.workunit.client.0.smithi117.stdout:test api_aio_pp on pid 39865 2024-05-15T12:44:45.531 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test api_aio_pp on pid 39865' 2024-05-15T12:44:45.531 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=39865 2024-05-15T12:44:45.531 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-15T12:44:45.531 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:44:45.531 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2024-05-15T12:44:45.531 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s api_io 2024-05-15T12:44:45.532 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' api_io' 2024-05-15T12:44:45.532 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo api_io 2024-05-15T12:44:45.532 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-05-15T12:44:45.532 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:44:45.532 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-05-15T12:44:45.532 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-05-15T12:44:45.532 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:44:45.532 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.532 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-05-15T12:44:45.532 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-05-15T12:44:45.532 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:44:45.532 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-05-15T12:44:45.532 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.533 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:44:45.533 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:44:45.533 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.533 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:44:45.533 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:44:45.533 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:44:45.533 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.533 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:44:45.533 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.533 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:44:45.533 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:44:45.534 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:44:45.534 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-05-15T12:44:45.534 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.534 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.534 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:44:45.534 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:44:45.534 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.534 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.534 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:44:45.534 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:44:45.534 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-05-15T12:44:45.534 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-05-15T12:44:45.534 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:45.534 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:45.534 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.535 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.535 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:44:45.535 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:44:45.535 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.535 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:44:45.535 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.535 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:44:45.535 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.535 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.535 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=api_io 2024-05-15T12:44:45.535 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=39874 2024-05-15T12:44:45.535 INFO:tasks.workunit.client.0.smithi117.stdout:test api_io on pid 39874 2024-05-15T12:44:45.539 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test api_io on pid 39874' 2024-05-15T12:44:45.539 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=39874 2024-05-15T12:44:45.540 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-15T12:44:45.540 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2024-05-15T12:44:45.540 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:44:45.540 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s api_io_pp 2024-05-15T12:44:45.540 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' api_io_pp' 2024-05-15T12:44:45.540 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo api_io_pp 2024-05-15T12:44:45.540 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-05-15T12:44:45.540 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:44:45.540 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-05-15T12:44:45.540 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-05-15T12:44:45.541 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:44:45.541 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.541 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-05-15T12:44:45.541 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-05-15T12:44:45.541 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:44:45.541 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:44:45.541 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:44:45.541 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:44:45.541 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:44:45.541 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.541 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:44:45.541 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:44:45.541 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-05-15T12:44:45.541 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.541 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:44:45.542 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:44:45.542 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.542 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:44:45.542 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:44:45.542 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-05-15T12:44:45.542 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.542 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.542 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:44:45.542 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:44:45.542 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.542 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:44:45.542 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:44:45.542 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-05-15T12:44:45.542 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.543 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-05-15T12:44:45.543 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.543 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.543 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:44:45.543 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:44:45.543 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.543 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:44:45.543 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.543 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:44:45.543 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.543 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:44:45.543 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.543 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:44:45.543 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.543 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:44:45.544 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:45.544 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:45.544 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.544 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:44:45.544 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.544 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:44:45.544 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.544 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=api_io_pp 2024-05-15T12:44:45.544 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=39886 2024-05-15T12:44:45.544 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test api_io_pp on pid 39886' 2024-05-15T12:44:45.545 INFO:tasks.workunit.client.0.smithi117.stdout:test api_io_pp on pid 39886 2024-05-15T12:44:45.549 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=39886 2024-05-15T12:44:45.550 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-15T12:44:45.550 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:44:45.550 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2024-05-15T12:44:45.550 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s api_asio 2024-05-15T12:44:45.550 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' api_asio' 2024-05-15T12:44:45.550 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo api_asio 2024-05-15T12:44:45.550 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-05-15T12:44:45.550 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:44:45.550 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-05-15T12:44:45.551 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-05-15T12:44:45.551 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:44:45.551 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.551 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-05-15T12:44:45.551 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:44:45.551 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:44:45.551 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:44:45.551 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:44:45.551 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.551 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:44:45.551 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.551 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:44:45.551 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:44:45.551 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:44:45.552 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:44:45.552 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.552 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.552 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:44:45.552 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.552 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:44:45.552 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:44:45.552 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.552 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:44:45.552 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-05-15T12:44:45.552 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:44:45.552 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.552 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.552 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:44:45.552 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.553 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:44:45.553 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-05-15T12:44:45.553 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:44:45.553 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:44:45.553 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:44:45.553 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-05-15T12:44:45.553 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:44:45.553 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.553 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:44:45.553 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.553 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:44:45.554 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.554 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:44:45.554 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.554 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:44:45.554 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:44:45.554 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-05-15T12:44:45.554 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.554 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:44:45.554 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.554 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:44:45.554 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-05-15T12:44:45.554 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:44:45.554 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-05-15T12:44:45.554 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-05-15T12:44:45.554 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.555 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.555 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:44:45.555 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:44:45.555 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.555 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:44:45.555 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.555 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:44:45.555 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:45.555 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:45.555 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.555 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.555 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:44:45.555 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.555 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:44:45.555 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.555 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=api_asio 2024-05-15T12:44:45.556 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=39899 2024-05-15T12:44:45.556 INFO:tasks.workunit.client.0.smithi117.stdout:test api_asio on pid 39899 2024-05-15T12:44:45.565 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test api_asio on pid 39899' 2024-05-15T12:44:45.565 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=39899 2024-05-15T12:44:45.565 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-15T12:44:45.565 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2024-05-15T12:44:45.565 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:44:45.565 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s api_list 2024-05-15T12:44:45.565 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:45.565 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:44:45.565 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:45.566 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:44:45.566 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' api_list' 2024-05-15T12:44:45.566 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:44:45.566 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:44:45.566 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo api_list 2024-05-15T12:44:45.566 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-05-15T12:44:45.566 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:44:45.566 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-05-15T12:44:45.566 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-05-15T12:44:45.566 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:44:45.566 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.567 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-05-15T12:44:45.567 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:44:45.567 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:44:45.567 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:44:45.567 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.567 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:44:45.567 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.567 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:44:45.567 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:44:45.567 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:44:45.567 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.567 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:44:45.567 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:44:45.567 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.567 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:44:45.567 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:44:45.567 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:44:45.567 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.568 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:44:45.568 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.568 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:44:45.568 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:44:45.568 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-05-15T12:44:45.568 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:44:45.568 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.568 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:44:45.568 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.568 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:44:45.568 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-05-15T12:44:45.568 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:44:45.568 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.568 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.568 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.568 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:44:45.568 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:44:45.569 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:44:45.569 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.569 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:44:45.569 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:44:45.569 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-05-15T12:44:45.569 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-05-15T12:44:45.569 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.569 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-05-15T12:44:45.569 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.569 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:44:45.569 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.569 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:44:45.569 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.569 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:44:45.570 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-05-15T12:44:45.570 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.570 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:44:45.570 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:44:45.570 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:44:45.570 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:44:45.570 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.570 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:44:45.570 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:45.570 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:44:45.570 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-05-15T12:44:45.570 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:45.570 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:44:45.570 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-05-15T12:44:45.571 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:44:45.571 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.571 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:44:45.571 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:45.571 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.571 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:44:45.571 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:45.571 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:45.571 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.571 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:45.571 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:44:45.571 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:44:45.571 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.571 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:44:45.571 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.571 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:44:45.571 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.572 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:44:45.572 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.572 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:44:45.572 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.572 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:44:45.572 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.572 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:44:45.572 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:44:45.572 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/39858/exe 2024-05-15T12:44:45.572 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=api_list 2024-05-15T12:44:45.572 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=39916 2024-05-15T12:44:45.572 INFO:tasks.workunit.client.0.smithi117.stdout:test api_list on pid 39916 2024-05-15T12:44:45.581 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test api_list on pid 39916' 2024-05-15T12:44:45.581 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=39916 2024-05-15T12:44:45.581 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-15T12:44:45.581 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:44:45.581 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2024-05-15T12:44:45.581 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s api_lock 2024-05-15T12:44:45.581 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:45.581 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:44:45.581 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:45.581 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:44:45.581 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:44:45.581 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' api_lock' 2024-05-15T12:44:45.581 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-05-15T12:44:45.582 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:44:45.582 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo api_lock 2024-05-15T12:44:45.582 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-05-15T12:44:45.582 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:44:45.582 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-05-15T12:44:45.582 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-05-15T12:44:45.582 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-05-15T12:44:45.582 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-05-15T12:44:45.582 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:44:45.582 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-05-15T12:44:45.582 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:44:45.582 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-05-15T12:44:45.582 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:44:45.582 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-05-15T12:44:45.583 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-05-15T12:44:45.583 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:44:45.583 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:44:45.583 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.583 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:44:45.583 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.583 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:44:45.583 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:44:45.583 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-05-15T12:44:45.583 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.583 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-05-15T12:44:45.583 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:44:45.583 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-05-15T12:44:45.583 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:44:45.583 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.583 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:44:45.584 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:44:45.584 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:44:45.584 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:44:45.584 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.584 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-05-15T12:44:45.584 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:44:45.584 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.584 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:44:45.584 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.584 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:44:45.584 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:44:45.584 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_rados_api_aio.log 2024-05-15T12:44:45.584 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:44:45.584 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.584 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:44:45.585 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:44:45.585 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:44:45.585 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.585 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-05-15T12:44:45.585 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:44:45.585 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:44:45.585 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.585 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-05-15T12:44:45.585 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:44:45.585 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.585 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:44:45.585 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.585 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:44:45.585 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2024-05-15T12:44:45.585 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-05-15T12:44:45.585 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.586 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:44:45.586 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:44:45.586 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:44:45.586 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.586 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:44:45.586 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.586 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:45.586 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:44:45.586 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.586 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:44:45.587 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:44:45.587 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-05-15T12:44:45.587 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:45.587 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-05-15T12:44:45.587 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:44:45.587 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:44:45.587 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-05-15T12:44:45.587 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.587 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:44:45.587 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.587 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:44:45.587 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:45.587 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:45.587 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.587 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:44:45.587 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.588 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:44:45.588 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:45.588 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.588 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:44:45.588 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.588 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:44:45.588 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.588 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:44:45.588 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:45.588 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:44:45.588 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.588 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:44:45.588 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-05-15T12:44:45.588 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.588 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:44:45.588 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.589 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:44:45.589 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:44:45.589 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.589 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:44:45.589 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.589 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ api_aio: /' 2024-05-15T12:44:45.589 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/39865/exe 2024-05-15T12:44:45.589 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=api_lock 2024-05-15T12:44:45.589 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=39937 2024-05-15T12:44:45.589 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test api_lock on pid 39937' 2024-05-15T12:44:45.589 INFO:tasks.workunit.client.0.smithi117.stdout:test api_lock on pid 39937 2024-05-15T12:44:45.598 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=39937 2024-05-15T12:44:45.598 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-15T12:44:45.598 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:44:45.598 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:45.598 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:44:45.598 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:45.598 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:44:45.598 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:44:45.598 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s api_lock_pp 2024-05-15T12:44:45.598 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-05-15T12:44:45.598 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' api_lock_pp' 2024-05-15T12:44:45.598 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2024-05-15T12:44:45.599 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:44:45.599 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:44:45.599 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:44:45.599 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:44:45.599 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.599 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:44:45.599 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.599 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:44:45.599 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:44:45.599 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.599 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:44:45.599 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-05-15T12:44:45.599 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.599 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:44:45.599 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-05-15T12:44:45.599 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-05-15T12:44:45.600 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.600 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-05-15T12:44:45.600 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-05-15T12:44:45.600 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-05-15T12:44:45.600 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:44:45.600 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.600 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:44:45.600 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-05-15T12:44:45.600 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-05-15T12:44:45.600 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:44:45.600 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:44:45.600 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-05-15T12:44:45.600 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:44:45.600 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:44:45.600 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.601 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:44:45.601 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-05-15T12:44:45.601 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:44:45.601 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-05-15T12:44:45.601 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-05-15T12:44:45.601 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:44:45.601 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.601 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:44:45.601 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:44:45.601 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.601 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-05-15T12:44:45.601 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:44:45.601 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:44:45.601 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.601 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-05-15T12:44:45.602 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:44:45.602 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:44:45.602 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:45.602 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:44:45.602 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:45.602 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_rados_api_aio_pp.log 2024-05-15T12:44:45.602 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:44:45.602 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo api_lock_pp 2024-05-15T12:44:45.602 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:44:45.602 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-05-15T12:44:45.602 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:44:45.602 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:45.602 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2024-05-15T12:44:45.602 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:44:45.602 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.603 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:44:45.603 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.603 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:44:45.603 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:45.603 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:44:45.603 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.603 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:44:45.603 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-05-15T12:44:45.603 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.603 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:44:45.603 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.604 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:44:45.604 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-05-15T12:44:45.604 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-05-15T12:44:45.604 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.604 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:44:45.604 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.604 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:44:45.604 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.604 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:44:45.604 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.604 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:44:45.604 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-05-15T12:44:45.604 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:44:45.604 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.604 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:44:45.604 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.605 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:44:45.605 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:44:45.605 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:44:45.605 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-05-15T12:44:45.605 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ api_aio_pp: /' 2024-05-15T12:44:45.605 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=api_lock_pp 2024-05-15T12:44:45.605 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.605 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:44:45.605 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.605 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:44:45.605 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=39954 2024-05-15T12:44:45.605 INFO:tasks.workunit.client.0.smithi117.stdout:test api_lock_pp on pid 39954 2024-05-15T12:44:45.616 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test api_lock_pp on pid 39954' 2024-05-15T12:44:45.616 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=39954 2024-05-15T12:44:45.616 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-15T12:44:45.616 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:44:45.616 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:44:45.616 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.616 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s api_misc 2024-05-15T12:44:45.616 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:44:45.616 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.617 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:44:45.617 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:45.617 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' api_misc' 2024-05-15T12:44:45.617 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:45.617 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:45.617 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.617 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:44:45.617 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.617 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:44:45.617 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.617 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:44:45.617 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:45.618 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:44:45.618 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:44:45.618 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/39874/exe 2024-05-15T12:44:45.618 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:44:45.618 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:44:45.618 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2024-05-15T12:44:45.618 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:44:45.618 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:44:45.618 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-05-15T12:44:45.618 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.618 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:44:45.618 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.618 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:44:45.618 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:44:45.619 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.619 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:44:45.619 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.619 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:44:45.619 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:44:45.619 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:44:45.619 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.619 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:44:45.619 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.619 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:44:45.619 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:44:45.619 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-05-15T12:44:45.619 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:45.619 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:44:45.619 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:45.620 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:44:45.620 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-05-15T12:44:45.620 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-05-15T12:44:45.620 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:45.620 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-05-15T12:44:45.620 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:45.620 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-05-15T12:44:45.620 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:44:45.620 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-05-15T12:44:45.620 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:44:45.620 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-05-15T12:44:45.620 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:44:45.620 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-05-15T12:44:45.620 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-05-15T12:44:45.621 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.621 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:44:45.621 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.621 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:44:45.621 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:44:45.621 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:44:45.621 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:44:45.621 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.621 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-05-15T12:44:45.621 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:44:45.621 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-05-15T12:44:45.621 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:44:45.621 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.621 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:44:45.621 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.622 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:44:45.622 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-05-15T12:44:45.622 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:44:45.622 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.622 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-05-15T12:44:45.622 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:44:45.622 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.622 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:44:45.622 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.622 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:44:45.622 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-05-15T12:44:45.622 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:44:45.622 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-05-15T12:44:45.622 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:44:45.622 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-05-15T12:44:45.623 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/39886/exe 2024-05-15T12:44:45.623 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-05-15T12:44:45.623 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2024-05-15T12:44:45.623 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.623 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:44:45.623 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.623 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:44:45.623 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:44:45.623 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:44:45.623 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:44:45.623 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:44:45.623 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.623 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:44:45.623 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.623 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:44:45.624 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.624 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:44:45.624 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:44:45.624 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-05-15T12:44:45.624 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ api_io: /' 2024-05-15T12:44:45.624 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.624 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:44:45.624 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.624 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:44:45.624 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-05-15T12:44:45.624 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-05-15T12:44:45.624 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.624 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:44:45.624 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.624 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:44:45.625 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.625 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.625 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:45.625 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:44:45.625 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:44:45.625 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:45.625 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:44:45.625 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:44:45.625 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:45.625 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:45.625 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo api_misc 2024-05-15T12:44:45.625 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.625 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:44:45.625 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.625 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:44:45.626 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.626 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_rados_api_io.log 2024-05-15T12:44:45.626 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-05-15T12:44:45.626 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:44:45.626 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=api_misc 2024-05-15T12:44:45.626 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=39979 2024-05-15T12:44:45.626 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test api_misc on pid 39979' 2024-05-15T12:44:45.626 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-05-15T12:44:45.627 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-05-15T12:44:45.627 INFO:tasks.workunit.client.0.smithi117.stdout:test api_misc on pid 39979 2024-05-15T12:44:45.638 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-05-15T12:44:45.638 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-05-15T12:44:45.638 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-05-15T12:44:45.638 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=39979 2024-05-15T12:44:45.639 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-05-15T12:44:45.639 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-15T12:44:45.639 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:44:45.639 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:44:45.639 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-05-15T12:44:45.639 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:44:45.639 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-05-15T12:44:45.639 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:44:45.639 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:44:45.639 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2024-05-15T12:44:45.639 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:44:45.639 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.639 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:44:45.639 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.639 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:44:45.640 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:44:45.640 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:44:45.640 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.640 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:44:45.640 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:44:45.640 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.640 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:44:45.640 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.640 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:44:45.640 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:44:45.640 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.640 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:44:45.640 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.640 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:44:45.640 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.641 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:44:45.641 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.641 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:44:45.641 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:44:45.641 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-05-15T12:44:45.641 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:44:45.641 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:44:45.641 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:45.641 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ api_io_pp: /' 2024-05-15T12:44:45.641 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:44:45.641 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:45.641 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:44:45.641 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-05-15T12:44:45.641 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2024-05-15T12:44:45.642 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:45.642 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:45.642 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:44:45.642 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_rados_api_io_pp.log 2024-05-15T12:44:45.642 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.642 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:44:45.642 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.642 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:44:45.643 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:44:45.643 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:44:45.643 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-05-15T12:44:45.643 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.643 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:44:45.643 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.643 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-05-15T12:44:45.643 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:44:45.643 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.643 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-05-15T12:44:45.643 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:44:45.643 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:44:45.643 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-05-15T12:44:45.643 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s api_misc_pp 2024-05-15T12:44:45.643 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' api_misc_pp' 2024-05-15T12:44:45.643 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:44:45.643 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-05-15T12:44:45.644 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.644 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:44:45.644 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.644 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:44:45.644 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:44:45.644 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.644 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:44:45.644 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.644 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:44:45.644 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:45.644 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:45.644 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.644 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:44:45.645 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.645 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:44:45.645 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-05-15T12:44:45.645 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.645 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:44:45.645 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:44:45.645 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:44:45.645 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo api_misc_pp 2024-05-15T12:44:45.645 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.645 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:44:45.645 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.645 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:44:45.645 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:44:45.646 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-05-15T12:44:45.646 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.646 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:44:45.646 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.646 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:44:45.646 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-05-15T12:44:45.646 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-05-15T12:44:45.646 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-05-15T12:44:45.646 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.646 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:44:45.646 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.647 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:44:45.647 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.647 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:44:45.647 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:44:45.647 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:44:45.647 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.647 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:44:45.647 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.647 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:44:45.647 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:44:45.647 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.648 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:44:45.648 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.648 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:44:45.648 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.648 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:44:45.648 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.648 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:44:45.648 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-05-15T12:44:45.648 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:45.648 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:44:45.648 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/39899/exe 2024-05-15T12:44:45.648 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:45.648 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:44:45.648 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:44:45.648 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=api_misc_pp 2024-05-15T12:44:45.648 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=39999 2024-05-15T12:44:45.649 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test api_misc_pp on pid 39999' 2024-05-15T12:44:45.649 INFO:tasks.workunit.client.0.smithi117.stdout:test api_misc_pp on pid 39999 2024-05-15T12:44:45.661 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=39999 2024-05-15T12:44:45.661 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-15T12:44:45.661 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-05-15T12:44:45.661 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:44:45.661 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-05-15T12:44:45.661 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:44:45.661 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-05-15T12:44:45.661 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-05-15T12:44:45.662 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s api_tier_pp 2024-05-15T12:44:45.662 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-05-15T12:44:45.662 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-05-15T12:44:45.662 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-05-15T12:44:45.662 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-05-15T12:44:45.662 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:44:45.662 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:44:45.662 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2024-05-15T12:44:45.662 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:44:45.662 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-05-15T12:44:45.662 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:44:45.662 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:45.662 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-05-15T12:44:45.662 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' api_tier_pp' 2024-05-15T12:44:45.663 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:44:45.663 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:44:45.663 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:45.663 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:44:45.663 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:44:45.663 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.663 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:44:45.663 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.663 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:44:45.663 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:44:45.663 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:44:45.663 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-05-15T12:44:45.663 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.663 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:44:45.663 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.664 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:44:45.664 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-05-15T12:44:45.664 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-05-15T12:44:45.664 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.664 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:44:45.664 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.664 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:44:45.664 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.664 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_rados_api_asio.log 2024-05-15T12:44:45.664 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:44:45.664 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:44:45.664 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:44:45.664 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.664 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:44:45.664 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.665 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:44:45.665 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:44:45.665 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.665 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:44:45.665 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-05-15T12:44:45.665 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:44:45.665 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-05-15T12:44:45.665 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:44:45.665 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.666 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-05-15T12:44:45.666 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:44:45.666 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:44:45.666 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:45.666 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:44:45.666 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:45.666 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:44:45.666 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:44:45.667 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:44:45.667 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:44:45.667 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:45.667 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:44:45.667 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:45.667 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:44:45.667 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-05-15T12:44:45.667 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:45.667 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-05-15T12:44:45.667 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:45.667 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:44:45.667 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.668 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:44:45.668 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.668 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:44:45.668 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo api_tier_pp 2024-05-15T12:44:45.668 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:44:45.668 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.668 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:44:45.668 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.668 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:44:45.668 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:45.668 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:44:45.668 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:45.668 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2024-05-15T12:44:45.668 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:44:45.669 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.669 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ api_asio: /' 2024-05-15T12:44:45.669 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:44:45.669 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.669 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:44:45.669 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:44:45.669 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.669 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:44:45.669 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-05-15T12:44:45.669 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.669 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:44:45.669 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:44:45.669 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:44:45.669 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-05-15T12:44:45.669 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.670 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:44:45.670 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.670 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:44:45.670 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:44:45.670 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.670 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:44:45.670 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.670 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:44:45.670 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:45.670 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:45.671 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.671 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:44:45.671 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-05-15T12:44:45.671 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.671 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:44:45.671 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.671 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/39916/exe 2024-05-15T12:44:45.671 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:44:45.671 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:44:45.671 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:44:45.671 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.671 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:44:45.672 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.672 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:44:45.672 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:44:45.672 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.672 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-05-15T12:44:45.672 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=api_tier_pp 2024-05-15T12:44:45.672 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-05-15T12:44:45.672 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40021 2024-05-15T12:44:45.672 INFO:tasks.workunit.client.0.smithi117.stdout:test api_tier_pp on pid 40021 2024-05-15T12:44:45.681 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test api_tier_pp on pid 40021' 2024-05-15T12:44:45.682 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=40021 2024-05-15T12:44:45.682 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-15T12:44:45.682 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:44:45.682 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-05-15T12:44:45.682 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-05-15T12:44:45.682 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-05-15T12:44:45.682 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-05-15T12:44:45.682 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-05-15T12:44:45.682 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:44:45.682 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-05-15T12:44:45.682 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:44:45.682 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-05-15T12:44:45.683 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s api_pool 2024-05-15T12:44:45.683 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' api_pool' 2024-05-15T12:44:45.683 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2024-05-15T12:44:45.683 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_rados_api_list.log 2024-05-15T12:44:45.683 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2024-05-15T12:44:45.683 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:44:45.683 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:44:45.683 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:44:45.683 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.683 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:44:45.683 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.683 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:44:45.683 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:44:45.684 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.684 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:44:45.684 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.684 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:44:45.684 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.684 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:44:45.684 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.684 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:44:45.684 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-05-15T12:44:45.684 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:44:45.684 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-05-15T12:44:45.684 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-05-15T12:44:45.684 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-05-15T12:44:45.684 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:44:45.685 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.685 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-05-15T12:44:45.685 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-05-15T12:44:45.685 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:44:45.685 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:44:45.685 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:44:45.685 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:44:45.685 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.685 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:44:45.685 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.685 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:44:45.685 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-05-15T12:44:45.685 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:44:45.685 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.685 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:44:45.685 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.686 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:44:45.686 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.686 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:44:45.686 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.686 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:44:45.686 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-05-15T12:44:45.686 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.686 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-05-15T12:44:45.686 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:44:45.687 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.687 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:44:45.687 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:44:45.687 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:44:45.687 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:44:45.687 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.687 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:44:45.687 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.687 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:44:45.687 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:44:45.687 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-05-15T12:44:45.687 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.687 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:44:45.687 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.687 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:44:45.687 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-05-15T12:44:45.687 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-05-15T12:44:45.688 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.688 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:44:45.688 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.688 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:44:45.688 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.688 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:44:45.688 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.688 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:44:45.688 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.688 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:44:45.688 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:45.689 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:45.689 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.689 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:44:45.689 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.689 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:44:45.689 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.689 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo api_pool 2024-05-15T12:44:45.689 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=api_pool 2024-05-15T12:44:45.689 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40039 2024-05-15T12:44:45.689 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test api_pool on pid 40039' 2024-05-15T12:44:45.689 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/39937/exe 2024-05-15T12:44:45.689 INFO:tasks.workunit.client.0.smithi117.stdout:test api_pool on pid 40039 2024-05-15T12:44:45.693 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=40039 2024-05-15T12:44:45.693 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-15T12:44:45.693 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:44:45.693 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s api_snapshots 2024-05-15T12:44:45.693 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' api_snapshots' 2024-05-15T12:44:45.693 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-05-15T12:44:45.693 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo api_snapshots 2024-05-15T12:44:45.693 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-05-15T12:44:45.693 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-05-15T12:44:45.694 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-05-15T12:44:45.694 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-05-15T12:44:45.694 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-05-15T12:44:45.694 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-05-15T12:44:45.694 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:44:45.694 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-05-15T12:44:45.694 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:44:45.694 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-05-15T12:44:45.694 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2024-05-15T12:44:45.694 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:44:45.694 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:44:45.694 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:44:45.694 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.694 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:44:45.694 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-05-15T12:44:45.695 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.695 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:44:45.695 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:44:45.695 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.695 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-05-15T12:44:45.695 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ api_list: /' 2024-05-15T12:44:45.695 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=api_snapshots 2024-05-15T12:44:45.695 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40057 2024-05-15T12:44:45.695 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test api_snapshots on pid 40057' 2024-05-15T12:44:45.695 INFO:tasks.workunit.client.0.smithi117.stdout:test api_snapshots on pid 40057 2024-05-15T12:44:45.707 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=40057 2024-05-15T12:44:45.707 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-15T12:44:45.707 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:44:45.707 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2024-05-15T12:44:45.707 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ api_lock: /' 2024-05-15T12:44:45.707 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s api_snapshots_pp 2024-05-15T12:44:45.707 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' api_snapshots_pp' 2024-05-15T12:44:45.707 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:44:45.708 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-05-15T12:44:45.708 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-05-15T12:44:45.708 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:44:45.708 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo api_snapshots_pp 2024-05-15T12:44:45.708 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.708 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-05-15T12:44:45.708 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-05-15T12:44:45.708 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:44:45.708 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-05-15T12:44:45.708 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.708 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:44:45.708 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.708 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:44:45.709 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:44:45.709 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.709 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:44:45.709 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.709 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:44:45.709 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:45.709 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:44:45.709 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:45.709 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_rados_api_lock.log 2024-05-15T12:44:45.709 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:45.709 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:44:45.709 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:44:45.709 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:45.709 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:44:45.710 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:44:45.710 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:44:45.710 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.710 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:44:45.710 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.710 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:44:45.710 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.710 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:44:45.710 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-05-15T12:44:45.710 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.710 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:44:45.710 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.710 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:44:45.711 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-05-15T12:44:45.711 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-05-15T12:44:45.711 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.711 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:44:45.711 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.711 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-05-15T12:44:45.711 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:44:45.711 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.711 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:44:45.711 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.711 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:44:45.711 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:45.711 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:44:45.712 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:45.712 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:44:45.712 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:44:45.712 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.712 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:44:45.712 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:44:45.712 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:44:45.712 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:44:45.712 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:44:45.712 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:45.712 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:44:45.712 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:45.712 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:44:45.713 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:44:45.713 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:44:45.713 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:44:45.713 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.713 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:44:45.713 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.713 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:44:45.713 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-05-15T12:44:45.713 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:44:45.713 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.713 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:44:45.713 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.714 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:44:45.714 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:44:45.714 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:44:45.714 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:45.714 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.714 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:44:45.714 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.714 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:44:45.714 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:45.714 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-05-15T12:44:45.714 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:44:45.714 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.714 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:44:45.715 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.715 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:44:45.715 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:44:45.715 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:44:45.715 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.715 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:44:45.715 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.715 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:44:45.715 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:44:45.715 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:45.715 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:44:45.716 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:45.716 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:44:45.716 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-05-15T12:44:45.716 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:45.716 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:45.716 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:44:45.716 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.716 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=api_snapshots_pp 2024-05-15T12:44:45.716 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:44:45.716 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.716 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:44:45.716 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40078 2024-05-15T12:44:45.717 INFO:tasks.workunit.client.0.smithi117.stdout:test api_snapshots_pp on pid 40078 2024-05-15T12:44:45.721 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test api_snapshots_pp on pid 40078' 2024-05-15T12:44:45.721 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=40078 2024-05-15T12:44:45.721 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-15T12:44:45.721 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:44:45.721 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:44:45.721 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.722 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:44:45.722 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.722 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:44:45.722 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:44:45.722 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2024-05-15T12:44:45.722 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s api_stat 2024-05-15T12:44:45.722 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' api_stat' 2024-05-15T12:44:45.722 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/39954/exe 2024-05-15T12:44:45.722 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-05-15T12:44:45.722 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:44:45.722 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo api_stat 2024-05-15T12:44:45.722 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/39979/exe 2024-05-15T12:44:45.723 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2024-05-15T12:44:45.723 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-05-15T12:44:45.723 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:44:45.723 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-05-15T12:44:45.723 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-05-15T12:44:45.723 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:44:45.723 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.723 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-05-15T12:44:45.723 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-05-15T12:44:45.723 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-05-15T12:44:45.723 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-05-15T12:44:45.723 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-05-15T12:44:45.723 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-05-15T12:44:45.723 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-05-15T12:44:45.724 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-05-15T12:44:45.724 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-05-15T12:44:45.724 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:44:45.724 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-05-15T12:44:45.724 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-05-15T12:44:45.724 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-05-15T12:44:45.724 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:44:45.724 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.724 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-05-15T12:44:45.724 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-05-15T12:44:45.724 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:44:45.724 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-05-15T12:44:45.725 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=api_stat 2024-05-15T12:44:45.725 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.725 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40096 2024-05-15T12:44:45.725 INFO:tasks.workunit.client.0.smithi117.stdout:test api_stat on pid 40096 2024-05-15T12:44:45.747 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test api_stat on pid 40096' 2024-05-15T12:44:45.747 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=40096 2024-05-15T12:44:45.748 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-05-15T12:44:45.748 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-15T12:44:45.748 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-05-15T12:44:45.748 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:44:45.748 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:44:45.748 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-05-15T12:44:45.748 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-05-15T12:44:45.748 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.748 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:44:45.748 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:44:45.748 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-05-15T12:44:45.748 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-05-15T12:44:45.748 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:44:45.749 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.749 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:44:45.749 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:44:45.749 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.749 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:44:45.749 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:45.749 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-05-15T12:44:45.749 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:45.749 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-05-15T12:44:45.749 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:44:45.749 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.749 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:44:45.749 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.750 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:44:45.750 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.750 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-05-15T12:44:45.750 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2024-05-15T12:44:45.750 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ api_misc: /' 2024-05-15T12:44:45.750 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_rados_api_misc.log 2024-05-15T12:44:45.750 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:44:45.750 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:44:45.750 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:44:45.750 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.750 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:44:45.751 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.751 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:44:45.751 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:44:45.751 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-05-15T12:44:45.751 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.751 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:44:45.751 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.751 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:44:45.751 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-05-15T12:44:45.751 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-05-15T12:44:45.751 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.751 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:44:45.752 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.752 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:44:45.752 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.752 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2024-05-15T12:44:45.752 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:44:45.752 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:44:45.752 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:44:45.752 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.752 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:44:45.752 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.752 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:44:45.753 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:44:45.753 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.753 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-05-15T12:44:45.753 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:44:45.753 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-05-15T12:44:45.753 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.753 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:44:45.753 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.753 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:44:45.753 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:44:45.753 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.753 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:44:45.754 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.754 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:44:45.754 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:45.754 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:45.754 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.754 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:44:45.754 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.754 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:44:45.754 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.754 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2024-05-15T12:44:45.754 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:44:45.754 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s api_stat_pp 2024-05-15T12:44:45.754 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' api_stat_pp' 2024-05-15T12:44:45.755 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ api_lock_pp: /' 2024-05-15T12:44:45.755 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:45.755 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:44:45.755 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:45.755 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:44:45.755 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:44:45.755 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-05-15T12:44:45.755 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-05-15T12:44:45.755 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:44:45.755 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.755 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-05-15T12:44:45.755 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-05-15T12:44:45.756 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:44:45.756 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-05-15T12:44:45.756 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.756 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:44:45.756 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.756 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:44:45.756 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:44:45.756 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.756 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:44:45.756 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.756 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:44:45.757 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:45.757 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:45.757 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.757 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:44:45.757 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.757 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:44:45.757 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.757 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:44:45.757 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:44:45.757 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:44:45.757 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.757 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:44:45.757 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.758 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:44:45.758 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:44:45.758 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.758 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_rados_api_lock_pp.log 2024-05-15T12:44:45.758 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:44:45.758 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:44:45.758 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:44:45.758 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.758 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:44:45.758 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.758 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:44:45.758 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:44:45.759 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-05-15T12:44:45.759 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.759 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:44:45.759 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.759 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:44:45.759 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-05-15T12:44:45.759 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-05-15T12:44:45.759 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.759 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:44:45.759 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.759 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:44:45.759 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.759 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:44:45.760 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:44:45.760 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:44:45.760 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:44:45.760 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:44:45.760 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:45.760 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:44:45.760 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:45.760 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:44:45.760 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-05-15T12:44:45.760 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:45.760 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:44:45.760 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:44:45.760 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:45.760 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:44:45.761 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.761 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:44:45.761 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.761 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:44:45.761 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:44:45.761 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.761 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:44:45.761 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.761 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:44:45.761 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.761 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:44:45.761 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:44:45.761 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.762 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:44:45.762 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:44:45.762 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-05-15T12:44:45.762 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.762 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:44:45.762 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.762 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:44:45.762 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-05-15T12:44:45.762 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-05-15T12:44:45.762 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.762 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:44:45.763 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.763 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:44:45.763 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.763 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:44:45.763 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:44:45.763 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:44:45.763 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.763 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:44:45.763 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.763 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:44:45.763 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:44:45.763 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-05-15T12:44:45.763 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.764 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:44:45.764 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.764 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:44:45.764 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-05-15T12:44:45.764 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-05-15T12:44:45.764 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.764 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:44:45.764 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.764 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:44:45.764 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.764 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo api_stat_pp 2024-05-15T12:44:45.764 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:44:45.765 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:44:45.765 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:44:45.765 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.765 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:44:45.765 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.765 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:44:45.765 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:44:45.765 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-05-15T12:44:45.765 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.765 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:44:45.765 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.765 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:44:45.765 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/39999/exe 2024-05-15T12:44:45.766 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.766 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:44:45.766 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.766 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:44:45.766 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-05-15T12:44:45.766 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:44:45.766 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:44:45.766 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:44:45.766 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.766 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:44:45.766 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.766 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:44:45.767 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:44:45.767 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.767 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-05-15T12:44:45.767 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=api_stat_pp 2024-05-15T12:44:45.767 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40165 2024-05-15T12:44:45.767 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test api_stat_pp on pid 40165' 2024-05-15T12:44:45.767 INFO:tasks.workunit.client.0.smithi117.stdout:test api_stat_pp on pid 40165 2024-05-15T12:44:45.781 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=40165 2024-05-15T12:44:45.781 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-15T12:44:45.781 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:44:45.781 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2024-05-15T12:44:45.781 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s api_watch_notify 2024-05-15T12:44:45.781 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:44:45.781 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:44:45.781 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:44:45.782 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' api_watch_notify' 2024-05-15T12:44:45.782 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.782 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:44:45.782 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.782 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:44:45.782 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:44:45.782 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-05-15T12:44:45.782 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.782 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-05-15T12:44:45.782 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-05-15T12:44:45.782 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-05-15T12:44:45.782 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-05-15T12:44:45.783 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-05-15T12:44:45.783 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-05-15T12:44:45.783 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:44:45.783 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:44:45.783 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-05-15T12:44:45.783 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-05-15T12:44:45.783 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:44:45.783 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-05-15T12:44:45.783 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-05-15T12:44:45.783 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:44:45.783 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.783 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-05-15T12:44:45.784 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_rados_api_misc_pp.log 2024-05-15T12:44:45.784 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:44:45.784 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:44:45.784 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:44:45.784 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.784 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:44:45.784 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.784 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:44:45.784 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:44:45.784 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.784 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2024-05-15T12:44:45.784 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:44:45.784 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:44:45.785 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:44:45.785 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.785 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:44:45.785 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.785 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:44:45.785 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:44:45.785 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.785 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:44:45.785 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.785 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:44:45.785 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.785 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:44:45.785 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.786 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:44:45.786 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-05-15T12:44:45.786 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:44:45.786 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:44:45.786 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:44:45.786 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.786 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:44:45.786 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.786 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:44:45.786 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:44:45.786 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.786 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:44:45.786 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.787 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:44:45.787 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.787 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:44:45.787 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.787 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:44:45.787 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-05-15T12:44:45.787 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo api_watch_notify 2024-05-15T12:44:45.787 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:45.787 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:44:45.787 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:45.787 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:44:45.787 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:44:45.787 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-05-15T12:44:45.788 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ api_misc_pp: /' 2024-05-15T12:44:45.788 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-05-15T12:44:45.788 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-05-15T12:44:45.788 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:44:45.788 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:44:45.788 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:44:45.788 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.788 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:44:45.788 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.788 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:44:45.788 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:44:45.788 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:44:45.788 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:44:45.789 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.789 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:44:45.789 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.789 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:44:45.789 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:44:45.789 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:44:45.789 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.789 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:44:45.789 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.789 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:44:45.789 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.789 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:44:45.789 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.790 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:44:45.790 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.790 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:44:45.790 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-05-15T12:44:45.790 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.790 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:44:45.790 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.790 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-05-15T12:44:45.790 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:44:45.790 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.790 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:44:45.790 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-05-15T12:44:45.791 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:44:45.791 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:45.791 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:44:45.791 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-05-15T12:44:45.791 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:45.791 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:44:45.791 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:44:45.791 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.791 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:44:45.791 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:44:45.791 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-05-15T12:44:45.791 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.792 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:44:45.792 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:44:45.792 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.792 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:44:45.792 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.792 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:44:45.792 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:45.792 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:45.792 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.792 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:44:45.792 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.792 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:44:45.792 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.792 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=api_watch_notify 2024-05-15T12:44:45.793 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40200 2024-05-15T12:44:45.793 INFO:tasks.workunit.client.0.smithi117.stdout:test api_watch_notify on pid 40200 2024-05-15T12:44:45.821 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test api_watch_notify on pid 40200' 2024-05-15T12:44:45.821 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=40200 2024-05-15T12:44:45.821 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-15T12:44:45.821 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:44:45.821 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2024-05-15T12:44:45.821 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:44:45.821 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:44:45.821 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:44:45.821 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.821 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:44:45.821 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.822 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:44:45.822 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:44:45.822 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-05-15T12:44:45.822 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.822 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:44:45.822 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.822 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:44:45.822 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-05-15T12:44:45.822 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-05-15T12:44:45.822 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.822 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:44:45.822 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.823 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:44:45.823 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.823 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:45.823 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:44:45.823 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:45.823 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:44:45.823 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:44:45.823 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:44:45.823 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-05-15T12:44:45.823 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-05-15T12:44:45.823 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:44:45.823 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.824 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-05-15T12:44:45.824 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-05-15T12:44:45.824 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:44:45.824 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:45.824 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:44:45.824 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s api_watch_notify_pp 2024-05-15T12:44:45.824 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:45.824 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:44:45.824 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:44:45.824 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' api_watch_notify_pp' 2024-05-15T12:44:45.824 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:44:45.824 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:44:45.824 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:44:45.824 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:45.825 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:44:45.825 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:45.825 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:44:45.825 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:44:45.825 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-05-15T12:44:45.825 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:45.825 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:45.825 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:44:45.825 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.825 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:44:45.825 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.825 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:44:45.826 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:44:45.826 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.826 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:44:45.826 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.826 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:44:45.826 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:44:45.826 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:45.826 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:44:45.826 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:45.826 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:44:45.826 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:44:45.826 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:44:45.827 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:44:45.827 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:44:45.827 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:44:45.827 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:45.827 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:44:45.827 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:45.827 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:44:45.827 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-05-15T12:44:45.827 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:45.827 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:45.827 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:44:45.827 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.827 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:44:45.828 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.828 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:44:45.828 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:44:45.828 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.828 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:44:45.828 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.828 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:44:45.828 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-05-15T12:44:45.828 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:44:45.828 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:44:45.828 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:44:45.828 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-05-15T12:44:45.829 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.829 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:44:45.829 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.829 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:44:45.829 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:44:45.829 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.829 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:44:45.829 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:44:45.829 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:44:45.829 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:44:45.829 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:45.829 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:44:45.830 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:45.830 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:44:45.830 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-05-15T12:44:45.830 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:45.830 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.830 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:44:45.830 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:45.830 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:44:45.830 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.830 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:44:45.830 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.830 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:44:45.831 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:44:45.831 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.831 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:44:45.831 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.831 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:44:45.831 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:45.831 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:44:45.831 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:45.831 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.831 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:44:45.831 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.831 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:44:45.832 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.832 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/40057/exe 2024-05-15T12:44:45.832 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/40021/exe 2024-05-15T12:44:45.832 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:44:45.832 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:44:45.832 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:44:45.832 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:45.832 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:44:45.832 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:45.832 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-05-15T12:44:45.832 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-05-15T12:44:45.832 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:44:45.833 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-05-15T12:44:45.833 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:45.833 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:45.833 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:44:45.833 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.833 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:44:45.833 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.833 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:44:45.833 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:44:45.833 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.833 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:44:45.833 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-05-15T12:44:45.833 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-05-15T12:44:45.834 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.834 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-05-15T12:44:45.834 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-05-15T12:44:45.834 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-05-15T12:44:45.834 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-05-15T12:44:45.834 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-05-15T12:44:45.834 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-05-15T12:44:45.834 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-05-15T12:44:45.834 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-05-15T12:44:45.834 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:44:45.834 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-05-15T12:44:45.834 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:44:45.834 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-05-15T12:44:45.835 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:44:45.835 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-05-15T12:44:45.835 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:44:45.835 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-05-15T12:44:45.835 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-05-15T12:44:45.835 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:44:45.835 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:44:45.835 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-05-15T12:44:45.835 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ api_tier_pp: /' 2024-05-15T12:44:45.835 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/40096/exe 2024-05-15T12:44:45.835 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ api_snapshots: /' 2024-05-15T12:44:45.835 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-05-15T12:44:45.836 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_rados_api_snapshots.log 2024-05-15T12:44:45.836 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_rados_api_tier_pp.log 2024-05-15T12:44:45.836 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-05-15T12:44:45.836 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-05-15T12:44:45.836 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-05-15T12:44:45.836 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-05-15T12:44:45.836 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-05-15T12:44:45.836 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-05-15T12:44:45.836 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:44:45.836 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-05-15T12:44:45.836 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:44:45.836 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-05-15T12:44:45.837 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ api_stat: /' 2024-05-15T12:44:45.837 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2024-05-15T12:44:45.837 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_rados_api_stat.log 2024-05-15T12:44:45.837 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:44:45.837 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:44:45.837 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:44:45.837 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:45.837 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:44:45.837 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:45.837 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:44:45.837 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-05-15T12:44:45.837 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:45.837 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:45.837 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:44:45.838 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.838 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:44:45.838 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.838 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:44:45.838 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:44:45.838 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.838 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:44:45.838 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.838 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:44:45.838 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:44:45.838 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:44:45.838 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2024-05-15T12:44:45.839 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/40078/exe 2024-05-15T12:44:45.839 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:44:45.839 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:44:45.839 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.839 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:44:45.839 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.839 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:44:45.839 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:44:45.839 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-05-15T12:44:45.839 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.839 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:44:45.839 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-05-15T12:44:45.839 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.840 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:44:45.840 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-05-15T12:44:45.840 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-05-15T12:44:45.840 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.840 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:44:45.840 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.840 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:44:45.840 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-05-15T12:44:45.840 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-05-15T12:44:45.840 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-05-15T12:44:45.840 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.840 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-05-15T12:44:45.840 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-05-15T12:44:45.841 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-05-15T12:44:45.841 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:44:45.841 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-05-15T12:44:45.841 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:44:45.841 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-05-15T12:44:45.841 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2024-05-15T12:44:45.841 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/40039/exe 2024-05-15T12:44:45.841 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo api_watch_notify_pp 2024-05-15T12:44:45.841 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-05-15T12:44:45.841 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-05-15T12:44:45.841 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-05-15T12:44:45.841 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-05-15T12:44:45.841 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-05-15T12:44:45.842 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-05-15T12:44:45.842 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-05-15T12:44:45.842 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:44:45.842 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-05-15T12:44:45.842 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:44:45.842 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-05-15T12:44:45.842 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:44:45.842 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:44:45.842 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:44:45.842 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2024-05-15T12:44:45.842 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.842 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:44:45.843 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.843 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:44:45.843 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:44:45.843 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.843 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:44:45.843 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:44:45.843 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:44:45.843 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.843 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:44:45.843 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.843 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:44:45.843 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:44:45.843 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.844 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-05-15T12:44:45.844 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:44:45.844 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:44:45.844 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:44:45.844 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.844 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:44:45.844 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.844 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:44:45.844 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:44:45.844 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.844 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:44:45.844 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.844 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:44:45.845 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.845 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:44:45.845 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.845 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:44:45.845 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-05-15T12:44:45.845 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2024-05-15T12:44:45.845 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ api_snapshots_pp: /' 2024-05-15T12:44:45.845 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2024-05-15T12:44:45.845 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_rados_api_pool.log 2024-05-15T12:44:45.845 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-05-15T12:44:45.845 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=api_watch_notify_pp 2024-05-15T12:44:45.845 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40287 2024-05-15T12:44:45.845 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test api_watch_notify_pp on pid 40287' 2024-05-15T12:44:45.846 INFO:tasks.workunit.client.0.smithi117.stdout:test api_watch_notify_pp on pid 40287 2024-05-15T12:44:45.852 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=40287 2024-05-15T12:44:45.852 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-15T12:44:45.853 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:44:45.853 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s api_cmd 2024-05-15T12:44:45.853 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' api_cmd' 2024-05-15T12:44:45.853 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ api_pool: /' 2024-05-15T12:44:45.853 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2024-05-15T12:44:45.853 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:44:45.853 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-05-15T12:44:45.853 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-05-15T12:44:45.853 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:44:45.853 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.853 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-05-15T12:44:45.853 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-05-15T12:44:45.853 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:44:45.854 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-05-15T12:44:45.854 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.854 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:44:45.854 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.854 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:44:45.854 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:44:45.854 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.854 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:44:45.854 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.854 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:44:45.854 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:45.854 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:45.855 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.855 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:44:45.855 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.855 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:44:45.855 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.855 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:44:45.855 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:44:45.855 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:44:45.855 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.855 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:44:45.855 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.855 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:44:45.855 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:44:45.856 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-05-15T12:44:45.856 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.856 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:44:45.856 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.856 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:44:45.856 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-05-15T12:44:45.856 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-05-15T12:44:45.856 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.856 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:44:45.856 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.856 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:44:45.856 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.857 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:44:45.857 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:44:45.857 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:44:45.857 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.857 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:44:45.857 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.857 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:44:45.857 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:44:45.857 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.858 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2024-05-15T12:44:45.858 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [----------] Global test environment set-up. 2024-05-15T12:44:45.858 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [----------] 12 tests from AsioRados 2024-05-15T12:44:45.858 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.682+0000 7fa39684b780 10 monclient: get_monmap_and_config 2024-05-15T12:44:45.858 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.682+0000 7fa39684b780 10 monclient: build_initial_monmap 2024-05-15T12:44:45.858 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.682+0000 7fa39684b780 10 monclient: monmap: 2024-05-15T12:44:45.858 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: epoch 0 2024-05-15T12:44:45.858 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:44:45.858 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: last_changed 0.000000 2024-05-15T12:44:45.858 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: created 0.000000 2024-05-15T12:44:45.858 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: min_mon_release 0 (unknown) 2024-05-15T12:44:45.858 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: election_strategy: 1 2024-05-15T12:44:45.859 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T12:44:45.859 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T12:44:45.859 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T12:44:45.859 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.859 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.683+0000 7fa39684b780 1 Processor -- start 2024-05-15T12:44:45.859 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.683+0000 7fa39684b780 1 -- start start 2024-05-15T12:44:45.859 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.683+0000 7fa39684b780 10 monclient: init 2024-05-15T12:44:45.859 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.684+0000 7fa39684b780 10 monclient: _reopen_session rank -1 2024-05-15T12:44:45.859 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.684+0000 7fa39684b780 10 monclient: _add_conns ranks=[2,1,0] 2024-05-15T12:44:45.859 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.684+0000 7fa39684b780 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x564ada801110 0x564ada8c8b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:44:45.859 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.684+0000 7fa39684b780 10 monclient(hunting): picked mon.noname-c con 0x564ada801110 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:44:45.859 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.684+0000 7fa39684b780 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564ada8d4330 0x564ada96b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:44:45.859 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.684+0000 7fa39684b780 10 monclient(hunting): picked mon.noname-b con 0x564ada8d4330 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:44:45.860 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.684+0000 7fa39684b780 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564ada8d4d70 0x564ada8d35d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:44:45.860 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.684+0000 7fa39684b780 10 monclient(hunting): picked mon.noname-a con 0x564ada8d4d70 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T12:44:45.860 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.684+0000 7fa39684b780 10 monclient(hunting): start opening mon connection 2024-05-15T12:44:45.860 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.684+0000 7fa39684b780 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ada801520 con 0x564ada8d4d70 2024-05-15T12:44:45.860 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.684+0000 7fa39684b780 10 monclient(hunting): start opening mon connection 2024-05-15T12:44:45.860 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.684+0000 7fa39684b780 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ada7b5b70 con 0x564ada8d4330 2024-05-15T12:44:45.860 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.684+0000 7fa39684b780 10 monclient(hunting): start opening mon connection 2024-05-15T12:44:45.860 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.684+0000 7fa39684b780 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x564ada8016a0 con 0x564ada801110 2024-05-15T12:44:45.860 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.684+0000 7fa39684b780 10 monclient(hunting): _renew_subs 2024-05-15T12:44:45.860 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.684+0000 7fa39684b780 10 monclient(hunting): authenticate will time out at 1448.926009s 2024-05-15T12:44:45.863 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:44:45.863 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:44:45.863 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:44:45.863 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.864 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:44:45.864 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.864 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:44:45.864 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:44:45.864 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.864 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:44:45.864 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.864 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:44:45.864 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.864 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:44:45.864 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.864 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:44:45.864 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-05-15T12:44:45.865 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo api_cmd 2024-05-15T12:44:45.865 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-05-15T12:44:45.865 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:45.865 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:44:45.865 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:45.865 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:44:45.865 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:44:45.865 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=api_cmd 2024-05-15T12:44:45.865 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40312 2024-05-15T12:44:45.865 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test api_cmd on pid 40312' 2024-05-15T12:44:45.865 INFO:tasks.workunit.client.0.smithi117.stdout: test api_cmd on pid 40312 2024-05-15T12:44:45.872 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=40312 2024-05-15T12:44:45.872 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-15T12:44:45.872 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:44:45.872 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-05-15T12:44:45.872 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2024-05-15T12:44:45.872 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:44:45.872 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-05-15T12:44:45.872 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-05-15T12:44:45.872 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:44:45.872 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.873 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-05-15T12:44:45.873 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:44:45.873 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-05-15T12:44:45.873 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s api_cmd_pp 2024-05-15T12:44:45.873 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:44:45.873 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-05-15T12:44:45.873 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.873 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:44:45.873 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.873 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:44:45.873 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:44:45.873 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' api_cmd_pp' 2024-05-15T12:44:45.873 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.874 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:44:45.874 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.874 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:44:45.874 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:45.874 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:45.874 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.874 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:44:45.874 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.874 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:44:45.874 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.874 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:44:45.874 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:44:45.874 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:44:45.875 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:45.875 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:44:45.875 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:45.875 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:44:45.875 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-05-15T12:44:45.875 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:45.875 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:45.875 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:44:45.875 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.875 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:44:45.875 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.875 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:44:45.875 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:44:45.876 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.876 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:44:45.876 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.876 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:44:45.876 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:44:45.876 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo api_cmd_pp 2024-05-15T12:44:45.876 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:45.876 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:44:45.876 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:45.876 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:44:45.876 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:44:45.876 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-05-15T12:44:45.877 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.708+0000 7fa3932b0640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564ada8d4330 0x564ada96b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:44:45.877 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.709+0000 7fa3932b0640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564ada8d4330 0x564ada96b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:52140/0 (socket says 172.21.15.117:52140) 2024-05-15T12:44:45.877 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.709+0000 7fa3932b0640 1 -- 172.21.15.117:0/383636253 learned_addr learned my addr 172.21.15.117:0/383636253 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:44:45.877 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.709+0000 7fa3932b0640 10 monclient(hunting): get_auth_request con 0x564ada8d4330 auth_method 0 2024-05-15T12:44:45.877 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.709+0000 7fa3932b0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:44:45.877 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.709+0000 7fa3932b0640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:44:45.877 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.709+0000 7fa3932b0640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:44:45.877 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.710+0000 7fa3932b0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:44:45.877 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.710+0000 7fa3932b0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:44:45.877 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.710+0000 7fa3932b0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:44:45.878 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.713+0000 7fa3932b0640 10 monclient(hunting): handle_auth_done global_id 4417 payload 274 2024-05-15T12:44:45.878 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.713+0000 7fa3932b0640 1 -- 172.21.15.117:0/383636253 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x564ada801110 msgr2=0x564ada8c8b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:44:45.878 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.713+0000 7fa3932b0640 1 --2- 172.21.15.117:0/383636253 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x564ada801110 0x564ada8c8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:44:45.878 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.713+0000 7fa3932b0640 1 -- 172.21.15.117:0/383636253 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564ada8d4d70 msgr2=0x564ada8d35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:44:45.878 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.713+0000 7fa3932b0640 1 --2- 172.21.15.117:0/383636253 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564ada8d4d70 0x564ada8d35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:44:45.878 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.713+0000 7fa3932b0640 10 monclient: _finish_hunting 0 2024-05-15T12:44:45.878 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.713+0000 7fa3932b0640 1 monclient: found mon.noname-b 2024-05-15T12:44:45.878 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.713+0000 7fa3932b0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:44:45.878 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.713+0000 7fa3932b0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2024-05-15T12:44:45.878 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.713+0000 7fa3932b0640 1 -- 172.21.15.117:0/383636253 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564ada96aef0 con 0x564ada8d4330 2024-05-15T12:44:45.878 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.720+0000 7fa3932b0640 1 --2- 172.21.15.117:0/383636253 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564ada8d4330 0x564ada96b410 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa388009750 tx=0x7fa388005a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18100e29abe3f07a server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:44:45.879 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.726+0000 7fa3922ae640 1 -- 172.21.15.117:0/383636253 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa388019070 con 0x564ada8d4330 2024-05-15T12:44:45.879 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.726+0000 7fa3922ae640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:44:45.879 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.726+0000 7fa3922ae640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:44:45.879 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.726+0000 7fa3922ae640 10 monclient: dump: 2024-05-15T12:44:45.879 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: epoch 1 2024-05-15T12:44:45.879 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:44:45.879 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:44:45.879 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:44:45.879 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: min_mon_release 19 (squid) 2024-05-15T12:44:45.879 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: election_strategy: 1 2024-05-15T12:44:45.879 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:44:45.879 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:44:45.879 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:44:45.880 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.880 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.726+0000 7fa3922ae640 10 monclient: _finish_auth 0 2024-05-15T12:44:45.880 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.726+0000 7fa3922ae640 10 monclient: _check_auth_tickets 2024-05-15T12:44:45.880 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.726+0000 7fa3922ae640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:44:45.880 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.726+0000 7fa3922ae640 1 -- 172.21.15.117:0/383636253 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa388008890 con 0x564ada8d4330 2024-05-15T12:44:45.880 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.726+0000 7fa3922ae640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:44:45.880 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.726+0000 7fa3922ae640 1 -- 172.21.15.117:0/383636253 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa388008c60 con 0x564ada8d4330 2024-05-15T12:44:45.880 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.726+0000 7fa3922ae640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:44:45.880 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.726+0000 7fa3922ae640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:44:45.880 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.726+0000 7fa3922ae640 10 monclient: dump: 2024-05-15T12:44:45.880 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: epoch 1 2024-05-15T12:44:45.880 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:44:45.880 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:44:45.881 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:44:45.881 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: min_mon_release 19 (squid) 2024-05-15T12:44:45.881 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: election_strategy: 1 2024-05-15T12:44:45.881 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:44:45.881 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:44:45.881 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:44:45.881 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.881 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.726+0000 7fa39684b780 5 monclient: authenticate success, global_id 4417 2024-05-15T12:44:45.881 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.726+0000 7fa39684b780 10 monclient: get_monmap_and_config success 2024-05-15T12:44:45.881 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.726+0000 7fa39684b780 4 set_mon_vals no callback set 2024-05-15T12:44:45.881 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.727+0000 7fa39684b780 10 monclient: shutdown 2024-05-15T12:44:45.881 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.727+0000 7fa39684b780 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:44:45.882 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.727+0000 7fa39684b780 1 -- 172.21.15.117:0/383636253 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564ada8d4330 msgr2=0x564ada96b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:44:45.885 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:44:45.885 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:44:45.885 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:44:45.885 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.885 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:44:45.886 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.886 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:44:45.886 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:44:45.886 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.886 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:44:45.886 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.886 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:44:45.886 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.886 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:44:45.886 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:44:45.886 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:44:45.886 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.887 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:44:45.887 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-05-15T12:44:45.887 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:44:45.887 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.887 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:44:45.887 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.887 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:44:45.887 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:44:45.887 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-05-15T12:44:45.887 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/40200/exe 2024-05-15T12:44:45.887 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.887 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:44:45.887 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.888 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:44:45.888 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=api_cmd_pp 2024-05-15T12:44:45.888 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-05-15T12:44:45.888 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-05-15T12:44:45.888 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.888 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40361 2024-05-15T12:44:45.888 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.727+0000 7fa39684b780 1 --2- 172.21.15.117:0/383636253 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564ada8d4330 0x564ada96b410 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa388009750 tx=0x7fa388005a00 ctest api_cmd_pp on pid 40361 2024-05-15T12:44:45.889 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test api_cmd_pp on pid 40361' 2024-05-15T12:44:45.890 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=40361 2024-05-15T12:44:45.890 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-15T12:44:45.890 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:44:45.890 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.890 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:44:45.890 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.890 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:44:45.890 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-05-15T12:44:45.890 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2024-05-15T12:44:45.890 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-05-15T12:44:45.890 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s api_service 2024-05-15T12:44:45.891 INFO:tasks.workunit.client.0.smithi117.stdout:omp rx=0 tx=0).stop 2024-05-15T12:44:45.891 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.727+0000 7fa39684b780 1 -- 172.21.15.117:0/383636253 shutdown_connections 2024-05-15T12:44:45.891 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.727+0000 7fa39684b780 1 --2- 172.21.15.117:0/383636253 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564ada8d4d70 0x564ada8d35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:44:45.891 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.727+0000 7fa39684b780 1 --2- 172.21.15.117:0/383636253 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564ada8d4330 0x564ada96b410 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:44:45.891 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.727+0000 7fa39684b780 1 --2- 172.21.15.117:0/383636253 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x564ada801110 0x564ada8c8b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:44:45.891 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.727+0000 7fa39684b780 1 -- 172.21.15.117:0/383636253 >> 172.21.15.117:0/383636253 conn(0x564ada843dc0 msgr2=0x564ada8c7540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:44:45.891 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.727+0000 7fa39684b780 1 -- 172.21.15.117:0/383636253 shutdown_connections 2024-05-15T12:44:45.891 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.728+0000 7fa39684b780 1 -- 172.21.15.117:0/383636253 wait complete. 2024-05-15T12:44:45.891 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.728+0000 7fa39684b780 10 monclient: build_initial_monmap 2024-05-15T12:44:45.892 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.728+0000 7fa39684b780 10 monclient: monmap: 2024-05-15T12:44:45.892 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: epoch 0 2024-05-15T12:44:45.892 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:44:45.892 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: last_changed 0.000000 2024-05-15T12:44:45.892 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: created 0.000000 2024-05-15T12:44:45.892 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: min_mon_release 0 (unknown) 2024-05-15T12:44:45.892 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: election_strategy: 1 2024-05-15T12:44:45.892 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T12:44:45.892 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T12:44:45.892 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T12:44:45.892 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.892 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.728+0000 7fa39684b780 1 librados: starting msgr at 2024-05-15T12:44:45.892 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.728+0000 7fa39684b780 1 librados: starting objecter 2024-05-15T12:44:45.893 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.728+0000 7fa39684b780 1 Processor -- start 2024-05-15T12:44:45.893 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.729+0000 7fa39684b780 1 -- start start 2024-05-15T12:44:45.893 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.729+0000 7fa39684b780 1 librados: setting wanted keys 2024-05-15T12:44:45.893 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.729+0000 7fa39684b780 1 librados: calling monclient init 2024-05-15T12:44:45.893 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.729+0000 7fa39684b780 10 monclient: init 2024-05-15T12:44:45.893 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.729+0000 7fa39684b780 10 monclient: _reopen_session rank -1 2024-05-15T12:44:45.893 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.729+0000 7fa39684b780 10 monclient: _add_conns ranks=[2,1,0] 2024-05-15T12:44:45.893 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.729+0000 7fa39684b780 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x564ada801110 0x564ada8460f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:44:45.893 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.729+0000 7fa39684b780 10 monclient(hunting): picked mon.noname-c con 0x564ada801110 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T12:44:45.893 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.729+0000 7fa39684b780 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564ada8d4330 0x564ada808470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:44:45.893 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.729+0000 7fa39684b780 10 monclient(hunting): picked mon.noname-b con 0x564ada8d4330 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T12:44:45.900 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:44:45.900 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' api_service' 2024-05-15T12:44:45.900 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-05-15T12:44:45.900 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-05-15T12:44:45.901 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-05-15T12:44:45.901 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-05-15T12:44:45.901 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-05-15T12:44:45.901 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-05-15T12:44:45.901 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-05-15T12:44:45.901 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:44:45.901 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-05-15T12:44:45.901 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:44:45.901 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-05-15T12:44:45.901 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:44:45.901 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:44:45.901 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-05-15T12:44:45.902 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2024-05-15T12:44:45.902 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:44:45.902 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:44:45.902 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_rados_api_watch_notify.log 2024-05-15T12:44:45.902 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:45.902 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:44:45.902 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-05-15T12:44:45.902 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:45.902 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:44:45.902 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-05-15T12:44:45.902 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:45.903 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:44:45.903 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.903 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-05-15T12:44:45.903 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:45.903 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:44:45.903 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.903 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:44:45.903 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.903 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-05-15T12:44:45.903 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:44:45.903 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:44:45.903 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.903 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:44:45.904 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:44:45.904 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.904 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:44:45.904 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:44:45.904 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-05-15T12:44:45.904 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.904 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:44:45.904 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.904 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:44:45.904 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:44:45.904 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.904 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:44:45.904 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.905 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:44:45.905 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ api_watch_notify: /' 2024-05-15T12:44:45.905 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:45.905 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:44:45.905 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:45.905 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:44:45.905 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:44:45.905 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:45.905 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:45.905 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.905 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:44:45.905 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.905 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:44:45.905 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.906 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/40287/exe 2024-05-15T12:44:45.906 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:44:45.906 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo api_service 2024-05-15T12:44:45.906 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.730+0000 7fa39684b780 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564ada8d4d70 0x564ada8089b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:44:45.906 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.730+0000 7fa39684b780 10 monclient(hunting): picked mon.noname-a con 0x564ada8d4d70 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T12:44:45.906 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.730+0000 7fa39684b780 10 monclient(hunting): start opening mon connection 2024-05-15T12:44:45.906 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.730+0000 7fa39684b780 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ada8ce5d0 con 0x564ada8d4d70 2024-05-15T12:44:45.906 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.730+0000 7fa39684b780 10 monclient(hunting): start opening mon connection 2024-05-15T12:44:45.906 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.730+0000 7fa39684b780 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ada8026e0 con 0x564ada8d4330 2024-05-15T12:44:45.907 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.730+0000 7fa39684b780 10 monclient(hunting): start opening mon connection 2024-05-15T12:44:45.907 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.730+0000 7fa39684b780 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x564ada96b950 con 0x564ada801110 2024-05-15T12:44:45.907 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.730+0000 7fa39684b780 10 monclient(hunting): _renew_subs 2024-05-15T12:44:45.907 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.730+0000 7fa39684b780 10 monclient(hunting): authenticate will time out at 1448.971848s 2024-05-15T12:44:45.907 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.732+0000 7fa3932b0640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564ada8d4330 0x564ada808470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:44:45.907 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.733+0000 7fa3932b0640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564ada8d4330 0x564ada808470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:52164/0 (socket says 172.21.15.117:52164) 2024-05-15T12:44:45.907 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.733+0000 7fa3932b0640 1 -- 172.21.15.117:0/4241893909 learned_addr learned my addr 172.21.15.117:0/4241893909 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T12:44:45.907 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.733+0000 7fa3932b0640 10 monclient(hunting): get_auth_request con 0x564ada8d4330 auth_method 0 2024-05-15T12:44:45.907 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.733+0000 7fa3932b0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:44:45.907 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.733+0000 7fa3932b0640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:44:45.907 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.733+0000 7fa3932b0640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:44:45.907 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.735+0000 7fa3932b0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:44:45.908 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.735+0000 7fa3932b0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:44:45.908 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.735+0000 7fa3932b0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:44:45.908 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.738+0000 7fa3932b0640 10 monclient(hunting): handle_auth_done global_id 4429 payload 824 2024-05-15T12:44:45.908 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.738+0000 7fa3932b0640 1 -- 172.21.15.117:0/4241893909 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x564ada801110 msgr2=0x564ada8460f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:44:45.908 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.738+0000 7fa3932b0640 1 --2- 172.21.15.117:0/4241893909 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x564ada801110 0x564ada8460f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:44:45.908 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=api_service 2024-05-15T12:44:45.908 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40381 2024-05-15T12:44:45.909 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.738+0000 7fa3932b0640 1 -- 172.21.15.117:0/4241893909 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] contest api_service on pid 40381 2024-05-15T12:44:45.922 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test api_service on pid 40381' 2024-05-15T12:44:45.922 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=40381 2024-05-15T12:44:45.922 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-15T12:44:45.922 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:44:45.922 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-05-15T12:44:45.922 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s api_service_pp 2024-05-15T12:44:45.922 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2024-05-15T12:44:45.922 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:44:45.922 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:44:45.922 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:44:45.922 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:44:45.923 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:44:45.923 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.923 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:44:45.923 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.923 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:44:45.923 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:44:45.923 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:44:45.923 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:45.923 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.923 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-05-15T12:44:45.923 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:44:45.923 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:45.924 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:44:45.924 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-05-15T12:44:45.924 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-05-15T12:44:45.924 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-05-15T12:44:45.924 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:45.924 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-05-15T12:44:45.924 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:45.924 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-05-15T12:44:45.924 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' api_service_pp' 2024-05-15T12:44:45.924 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-05-15T12:44:45.924 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:44:45.924 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:44:45.924 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.925 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-05-15T12:44:45.925 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:44:45.925 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.925 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:44:45.925 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:44:45.925 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:44:45.925 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-05-15T12:44:45.925 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.925 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:44:45.925 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.925 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:44:45.925 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:44:45.926 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-05-15T12:44:45.926 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:44:45.926 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-05-15T12:44:45.926 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:44:45.926 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.926 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-05-15T12:44:45.926 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-05-15T12:44:45.926 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2024-05-15T12:44:45.926 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:44:45.926 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-05-15T12:44:45.926 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.927 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:44:45.927 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.927 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:44:45.927 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:44:45.927 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.927 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:44:45.927 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:44:45.927 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:44:45.927 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.927 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:44:45.927 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:45.927 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:44:45.928 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.928 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:44:45.928 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.928 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:44:45.928 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2024-05-15T12:44:45.928 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:45.928 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:44:45.928 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.928 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-05-15T12:44:45.928 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:44:45.928 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.928 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.929 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:44:45.929 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:44:45.929 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.929 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:44:45.929 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.929 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-05-15T12:44:45.929 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-05-15T12:44:45.929 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.929 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:44:45.929 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.929 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:44:45.929 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.929 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:44:45.929 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2024-05-15T12:44:45.930 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:44:45.930 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:44:45.930 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.930 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:44:45.930 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.930 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:44:45.930 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:44:45.930 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.930 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:44:45.930 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.930 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:44:45.931 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo api_service_pp 2024-05-15T12:44:45.931 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.931 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:44:45.931 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.931 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:44:45.931 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-05-15T12:44:45.931 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-05-15T12:44:45.931 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=api_service_pp 2024-05-15T12:44:45.931 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40403 2024-05-15T12:44:45.931 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test api_service_pp on pid 40403' 2024-05-15T12:44:45.932 INFO:tasks.workunit.client.0.smithi117.stdout:test api_service_pp on pid 40403 2024-05-15T12:44:45.948 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=40403 2024-05-15T12:44:45.948 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-15T12:44:45.948 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:44:45.948 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s api_c_write_operations 2024-05-15T12:44:45.949 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' api_c_write_operations' 2024-05-15T12:44:45.949 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo api_c_write_operations 2024-05-15T12:44:45.949 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2024-05-15T12:44:45.949 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:44:45.949 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-05-15T12:44:45.949 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:44:45.949 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:44:45.949 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.949 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:44:45.949 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.949 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:44:45.949 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:44:45.949 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.950 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:44:45.950 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-05-15T12:44:45.950 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-05-15T12:44:45.950 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:44:45.950 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.950 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-05-15T12:44:45.950 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-05-15T12:44:45.950 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:44:45.950 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-05-15T12:44:45.950 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:44:45.950 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:44:45.951 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/40165/exe 2024-05-15T12:44:45.951 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.951 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:44:45.951 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.951 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:44:45.951 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.951 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:44:45.951 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:44:45.951 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.951 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:45.951 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:44:45.951 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.951 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:44:45.952 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:44:45.952 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.952 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:44:45.952 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:45.952 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:44:45.952 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:45.952 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:44:45.952 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:44:45.952 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:44:45.952 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-05-15T12:44:45.952 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.952 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:44:45.953 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.953 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:44:45.953 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:45.953 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-05-15T12:44:45.953 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-05-15T12:44:45.954 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.954 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:44:45.954 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.954 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:44:45.954 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.954 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.954 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:44:45.954 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.954 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:44:45.955 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.955 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:44:45.955 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-05-15T12:44:45.955 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-05-15T12:44:45.955 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-05-15T12:44:45.955 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-05-15T12:44:45.955 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-05-15T12:44:45.955 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-05-15T12:44:45.955 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-05-15T12:44:45.955 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:44:45.955 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:44:45.955 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:44:45.955 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:44:45.955 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-05-15T12:44:45.956 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:44:45.956 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:45.956 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:44:45.956 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:45.956 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:44:45.956 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-05-15T12:44:45.956 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:45.956 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-05-15T12:44:45.956 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:45.956 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:44:45.956 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.956 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:44:45.956 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.956 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:44:45.956 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:44:45.956 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.957 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:44:45.957 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.957 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ api_stat_pp: /' 2024-05-15T12:44:45.957 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:44:45.957 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:44:45.957 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:44:45.957 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:44:45.957 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:44:45.957 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_rados_api_stat_pp.log 2024-05-15T12:44:45.957 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.957 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:44:45.957 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.957 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:44:45.957 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:44:45.957 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-05-15T12:44:45.957 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.958 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:44:45.958 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.958 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:44:45.958 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-05-15T12:44:45.958 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-05-15T12:44:45.958 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.958 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:44:45.958 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.958 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:44:45.958 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.958 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2024-05-15T12:44:45.958 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/40312/exe 2024-05-15T12:44:45.958 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:44:45.958 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:44:45.958 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:44:45.959 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.959 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:44:45.959 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.959 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:44:45.959 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:44:45.959 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.959 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:44:45.959 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.959 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:44:45.959 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.959 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:44:45.959 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.959 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:44:45.959 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-05-15T12:44:45.959 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-05-15T12:44:45.960 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-05-15T12:44:45.960 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:44:45.960 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:44:45.960 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:44:45.960 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.960 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:44:45.960 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.960 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:44:45.960 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:44:45.960 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.960 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-05-15T12:44:45.960 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-05-15T12:44:45.960 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-05-15T12:44:45.960 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-05-15T12:44:45.960 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-05-15T12:44:45.961 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-05-15T12:44:45.961 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:44:45.961 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:45.961 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:44:45.961 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-05-15T12:44:45.961 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:44:45.961 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-05-15T12:44:45.961 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:45.961 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:44:45.961 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:44:45.961 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-05-15T12:44:45.961 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2024-05-15T12:44:45.962 INFO:tasks.workunit.client.0.smithi117.stdout:n(0x564ada8d4d70 msgr2=0x564ada8089b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:44:45.962 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.738+0000 7fa3932b0640 1 --2- 172.21.15.117:0/4241893909 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564ada8d4d70 0x564ada8089b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:44:45.962 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.738+0000 7fa3932b0640 10 monclient: _finish_hunting 0 2024-05-15T12:44:45.962 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.738+0000 7fa3932b0640 1 monclient: found mon.noname-b 2024-05-15T12:44:45.962 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.738+0000 7fa3932b0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:44:45.962 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.738+0000 7fa3932b0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2024-05-15T12:44:45.962 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.738+0000 7fa3932b0640 1 -- 172.21.15.117:0/4241893909 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564ada8d0cb0 con 0x564ada8d4330 2024-05-15T12:44:45.962 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.744+0000 7fa3932b0640 1 --2- 172.21.15.117:0/4241893909 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564ada8d4330 0x564ada808470 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa388000c00 tx=0x7fa388003020 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:44:45.962 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.744+0000 7fa3912ac640 10 client.?.objecter ms_handle_connect 0x564ada8d4330 2024-05-15T12:44:45.962 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.744+0000 7fa3912ac640 10 client.?.objecter resend_mon_ops 2024-05-15T12:44:45.962 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.765+0000 7fa3912ac640 1 -- 172.21.15.117:0/4241893909 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa388019030 con 0x564ada8d4330 2024-05-15T12:44:45.962 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.765+0000 7fa3912ac640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:44:45.963 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.765+0000 7fa3912ac640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:44:45.963 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.765+0000 7fa3912ac640 10 monclient: dump: 2024-05-15T12:44:45.963 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: epoch 1 2024-05-15T12:44:45.963 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:44:45.963 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:44:45.963 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:44:45.963 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: min_mon_release 19 (squid) 2024-05-15T12:44:45.963 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: election_strategy: 1 2024-05-15T12:44:45.963 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:44:45.963 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:44:45.963 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:44:45.963 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.963 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.765+0000 7fa3912ac640 10 monclient: _finish_auth 0 2024-05-15T12:44:45.963 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.765+0000 7fa3912ac640 10 monclient: _check_auth_tickets 2024-05-15T12:44:45.963 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.765+0000 7fa3912ac640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:44:45.964 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.765+0000 7fa3912ac640 1 -- 172.21.15.117:0/4241893909 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa388024070 con 0x564ada8d4330 2024-05-15T12:44:45.964 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.765+0000 7fa3912ac640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:44:45.964 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.765+0000 7fa3912ac640 1 -- 172.21.15.117:0/4241893909 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa388014890 con 0x564ada8d4330 2024-05-15T12:44:45.964 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.765+0000 7fa3912ac640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:44:45.964 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.765+0000 7fa3912ac640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:44:45.968 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_rados_api_cmd.log 2024-05-15T12:44:45.968 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:44:45.968 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:44:45.968 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:44:45.968 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:44:45.968 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:45.968 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ api_cmd: /' 2024-05-15T12:44:45.968 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:44:45.968 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:45.968 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:44:45.968 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-05-15T12:44:45.968 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:45.969 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:45.969 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:44:45.969 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.969 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:44:45.969 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.969 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:44:45.969 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:44:45.969 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.969 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:44:45.969 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.969 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:44:45.969 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:44:45.969 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:44:45.969 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:44:45.969 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.970 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:44:45.970 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.970 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:44:45.970 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:44:45.970 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:44:45.970 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.970 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:44:45.970 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.970 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:44:45.970 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.970 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:44:45.970 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.970 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:44:45.970 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-05-15T12:44:45.970 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=api_c_write_operations 2024-05-15T12:44:45.971 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40456 2024-05-15T12:44:45.971 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test api_c_write_operations on pid 40456' 2024-05-15T12:44:45.971 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.765+0000 7fa3912ac640 10 montest api_c_write_operations on pid 40456 2024-05-15T12:44:45.976 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=40456 2024-05-15T12:44:45.976 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-15T12:44:45.976 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:44:45.976 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2024-05-15T12:44:45.976 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:44:45.976 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:44:45.976 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:44:45.976 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.977 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:44:45.977 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.977 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:44:45.977 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:44:45.977 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.977 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s api_c_read_operations 2024-05-15T12:44:45.977 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' api_c_read_operations' 2024-05-15T12:44:45.977 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:44:45.977 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-05-15T12:44:45.977 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-05-15T12:44:45.977 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:44:45.977 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.977 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-05-15T12:44:45.977 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo api_c_read_operations 2024-05-15T12:44:45.977 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-05-15T12:44:45.977 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:44:45.978 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-05-15T12:44:45.978 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.978 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:44:45.978 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.978 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:44:45.978 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:44:45.978 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-05-15T12:44:45.978 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.978 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:44:45.978 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.978 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:44:45.978 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:45.978 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:45.978 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.979 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:44:45.979 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.979 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:44:45.979 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.979 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-05-15T12:44:45.979 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/40361/exe 2024-05-15T12:44:45.979 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:45.979 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:44:45.979 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:45.979 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:44:45.979 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:44:45.979 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-05-15T12:44:45.979 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-05-15T12:44:45.979 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-05-15T12:44:45.979 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-05-15T12:44:45.979 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-05-15T12:44:45.980 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=api_c_read_operations 2024-05-15T12:44:45.980 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-05-15T12:44:45.980 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-05-15T12:44:45.980 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:44:45.980 INFO:tasks.workunit.client.0.smithi117.stdout:test api_c_read_operations on pid 40470 2024-05-15T12:44:45.993 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40470 2024-05-15T12:44:45.993 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test api_c_read_operations on pid 40470' 2024-05-15T12:44:45.993 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=40470 2024-05-15T12:44:45.993 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-15T12:44:45.993 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:44:45.993 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-05-15T12:44:45.993 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:44:45.993 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-05-15T12:44:45.993 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2024-05-15T12:44:45.994 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2024-05-15T12:44:45.994 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ api_cmd_pp: /' 2024-05-15T12:44:45.994 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:44:45.994 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:44:45.994 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:44:45.994 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.994 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:44:45.994 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.994 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:44:45.994 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:44:45.994 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-05-15T12:44:45.994 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.994 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:44:45.994 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.995 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:44:45.995 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-05-15T12:44:45.995 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-05-15T12:44:45.995 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.995 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:44:45.995 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.995 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:44:45.995 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:44:45.995 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:44:45.995 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:44:45.995 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.995 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.995 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:44:45.995 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-05-15T12:44:45.995 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-05-15T12:44:45.995 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:44:45.996 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.996 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-05-15T12:44:45.996 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:44:45.996 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.996 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:44:45.996 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:44:45.996 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.996 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:44:45.996 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.996 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:44:45.996 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.996 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:44:45.996 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.996 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:44:45.996 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-05-15T12:44:45.996 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-05-15T12:44:45.997 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:44:45.997 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-05-15T12:44:45.997 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.997 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:44:45.997 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.997 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:44:45.997 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:44:45.997 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.997 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:44:45.997 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.997 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:44:45.997 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:45.997 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:45.997 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.997 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:44:45.997 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.997 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:44:45.997 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.998 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:44:45.998 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:44:45.998 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:44:45.998 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.998 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:44:45.998 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.998 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:44:45.998 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:44:45.998 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:44:45.998 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:45.998 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:44:45.998 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:44:45.998 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:44:45.998 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2024-05-15T12:44:45.999 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:45.999 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:44:45.999 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:44:45.999 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:44:45.999 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:44:45.999 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:45.999 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:44:45.999 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:45.999 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:45.999 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:44:45.999 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-05-15T12:44:45.999 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:44:45.999 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:45.999 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:45.999 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:44:46.000 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:44:46.000 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.000 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:44:46.000 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.000 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:44:46.000 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-05-15T12:44:46.000 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:44:46.000 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.000 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.000 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:44:46.000 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.000 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:44:46.000 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:44:46.000 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:44:46.000 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.001 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:44:46.001 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-05-15T12:44:46.001 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-05-15T12:44:46.001 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.001 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:44:46.001 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.001 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:44:46.001 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:46.001 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-05-15T12:44:46.001 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s api_cls_remote_reads 2024-05-15T12:44:46.001 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' api_cls_remote_reads' 2024-05-15T12:44:46.001 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/40403/exe 2024-05-15T12:44:46.001 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-05-15T12:44:46.001 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-05-15T12:44:46.001 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo api_cls_remote_reads 2024-05-15T12:44:46.001 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-05-15T12:44:46.002 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-05-15T12:44:46.002 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-05-15T12:44:46.002 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-05-15T12:44:46.002 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-05-15T12:44:46.002 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-05-15T12:44:46.002 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:44:46.002 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-05-15T12:44:46.002 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:44:46.002 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-05-15T12:44:46.002 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ api_service_pp: /' 2024-05-15T12:44:46.002 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:44:46.002 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:44:46.002 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:44:46.002 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.002 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:44:46.002 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.003 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:44:46.003 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:44:46.003 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:46.003 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2024-05-15T12:44:46.003 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=api_cls_remote_reads 2024-05-15T12:44:46.003 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40517 2024-05-15T12:44:46.003 INFO:tasks.workunit.client.0.smithi117.stdout:test api_cls_remote_reads on pid 40517 2024-05-15T12:44:46.015 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test api_cls_remote_reads on pid 40517' 2024-05-15T12:44:46.015 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=40517 2024-05-15T12:44:46.015 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-15T12:44:46.015 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:44:46.015 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cls_remote_reads.xml 2>&1 | tee ceph_test_rados_api_cls_remote_reads.log | sed "s/^/ api_cls_remote_reads: /"' 2024-05-15T12:44:46.015 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_rados_api_service_pp.log 2024-05-15T12:44:46.015 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:44:46.015 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-05-15T12:44:46.015 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-05-15T12:44:46.016 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:44:46.016 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.016 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-05-15T12:44:46.016 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:44:46.016 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:44:46.016 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:44:46.016 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.016 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:44:46.016 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.016 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:44:46.016 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:44:46.016 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.016 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:44:46.016 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.016 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:44:46.017 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.017 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:44:46.017 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.017 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:44:46.017 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-05-15T12:44:46.017 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-05-15T12:44:46.017 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:44:46.017 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-05-15T12:44:46.017 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.017 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:44:46.017 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.017 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:44:46.017 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:44:46.017 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.017 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:44:46.018 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.018 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:44:46.018 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:46.018 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:46.018 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.018 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:44:46.018 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.018 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:44:46.018 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:46.018 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s list_parallel 2024-05-15T12:44:46.018 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' list_parallel' 2024-05-15T12:44:46.018 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-05-15T12:44:46.018 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:46.018 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:44:46.019 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:46.019 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:44:46.019 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:44:46.019 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:46.019 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:44:46.019 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:46.019 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:44:46.019 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:44:46.019 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:44:46.019 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:44:46.019 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:44:46.019 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:44:46.019 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:46.019 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:44:46.019 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:46.020 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:44:46.020 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-05-15T12:44:46.020 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:46.020 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:44:46.020 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:44:46.020 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:44:46.020 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.020 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:44:46.020 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.020 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:44:46.020 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:44:46.020 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.020 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:44:46.020 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.021 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:44:46.021 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:46.021 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:44:46.021 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.021 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:44:46.021 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.021 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:44:46.021 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-05-15T12:44:46.021 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo list_parallel 2024-05-15T12:44:46.021 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.021 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:44:46.021 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.021 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:44:46.021 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:44:46.021 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.022 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:44:46.022 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.022 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:44:46.022 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:44:46.022 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-05-15T12:44:46.022 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:44:46.022 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:44:46.022 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:44:46.022 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:44:46.022 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:46.023 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:44:46.023 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:46.023 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:44:46.023 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-05-15T12:44:46.023 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:46.023 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:46.023 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:44:46.023 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.023 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-05-15T12:44:46.023 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:44:46.024 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.024 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:44:46.024 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:44:46.024 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.024 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:44:46.024 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.024 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:44:46.024 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:44:46.024 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/40381/exe 2024-05-15T12:44:46.024 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-05-15T12:44:46.024 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:44:46.024 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:46.024 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:44:46.024 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:46.024 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:44:46.025 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:44:46.025 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:44:46.025 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=list_parallel 2024-05-15T12:44:46.025 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:44:46.025 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.025 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40545 2024-05-15T12:44:46.025 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test list_parallel on pid 40545' 2024-05-15T12:44:46.025 INFO:tasks.workunit.client.0.smithi117.stdout:test list_parallel on pid 40545 2024-05-15T12:44:46.031 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=40545 2024-05-15T12:44:46.031 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-15T12:44:46.031 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:44:46.031 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:44:46.031 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.031 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:44:46.031 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2024-05-15T12:44:46.031 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:44:46.031 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-05-15T12:44:46.031 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.031 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:44:46.031 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.031 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:44:46.031 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-05-15T12:44:46.032 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-05-15T12:44:46.032 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.032 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:44:46.032 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.032 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:44:46.032 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:46.032 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-05-15T12:44:46.032 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-05-15T12:44:46.032 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-05-15T12:44:46.032 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-05-15T12:44:46.032 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:44:46.032 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-05-15T12:44:46.032 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-05-15T12:44:46.032 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-05-15T12:44:46.033 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-05-15T12:44:46.033 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:44:46.033 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.033 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-05-15T12:44:46.033 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:44:46.033 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-05-15T12:44:46.033 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s open_pools_parallel 2024-05-15T12:44:46.033 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:44:46.033 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-05-15T12:44:46.033 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' open_pools_parallel' 2024-05-15T12:44:46.033 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2024-05-15T12:44:46.033 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-05-15T12:44:46.033 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:44:46.033 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-05-15T12:44:46.033 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.033 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:44:46.034 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.034 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:44:46.034 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:44:46.034 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.034 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:44:46.034 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.034 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:44:46.034 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:46.034 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:46.034 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.034 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:44:46.034 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.034 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:44:46.034 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:46.034 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/40456/exe 2024-05-15T12:44:46.034 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-05-15T12:44:46.035 INFO:tasks.workunit.client.0.smithi117.stdout:client: dump: 2024-05-15T12:44:46.035 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: epoch 1 2024-05-15T12:44:46.035 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:44:46.035 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T12:44:46.035 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: created 2024-05-15T12:41:21.922077+0000 2024-05-15T12:44:46.035 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: min_mon_release 19 (squid) 2024-05-15T12:44:46.035 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: election_strategy: 1 2024-05-15T12:44:46.035 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T12:44:46.035 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T12:44:46.035 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T12:44:46.035 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:46.036 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.765+0000 7fa39684b780 5 monclient: authenticate success, global_id 4429 2024-05-15T12:44:46.036 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.765+0000 7fa39684b780 10 monclient: _renew_subs 2024-05-15T12:44:46.036 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.765+0000 7fa39684b780 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:44:46.036 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.765+0000 7fa39684b780 1 -- 172.21.15.117:0/4241893909 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x564ada802860 con 0x564ada8d4330 2024-05-15T12:44:46.036 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.765+0000 7fa39684b780 10 client.4429.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:44:46.036 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.765+0000 7fa39684b780 10 monclient: _renew_subs 2024-05-15T12:44:46.036 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.765+0000 7fa39684b780 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:44:46.036 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.765+0000 7fa39684b780 1 -- 172.21.15.117:0/4241893909 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564ada8d3d50 con 0x564ada8d4330 2024-05-15T12:44:46.036 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.765+0000 7fa39684b780 1 librados: init done 2024-05-15T12:44:46.036 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.765+0000 7fa39684b780 10 librados: wait_for_osdmap waiting 2024-05-15T12:44:46.036 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.765+0000 7fa391aad640 4 set_mon_vals no callback set 2024-05-15T12:44:46.036 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.812+0000 7fa3912ac640 1 -- 172.21.15.117:0/4241893909 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fa388014a30 con 0x564ada8d4330 2024-05-15T12:44:46.036 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.812+0000 7fa3912ac640 1 --2- 172.21.15.117:0/4241893909 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fa38006b0d0 0x7fa38006d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:44:46.036 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.813+0000 7fa393ab1640 1 --2- 172.21.15.117:0/4241893909 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fa38006b0d0 0x7fa38006d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:44:46.036 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.813+0000 7fa393ab1640 10 monclient: get_auth_request con 0x7fa38006b0d0 auth_method 0 2024-05-15T12:44:46.036 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.814+0000 7fa393ab1640 1 --2- 172.21.15.117:0/4241893909 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fa38006b0d0 0x7fa38006d590 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa384054000 tx=0x7fa38404c070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:44:46.037 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.814+0000 7fa3912ac640 10 client.4429.objecter ms_handle_connect 0x7fa38006b0d0 2024-05-15T12:44:46.037 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.827+0000 7fa3912ac640 1 -- 172.21.15.117:0/4241893909 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fa388057400 con 0x564ada8d4330 2024-05-15T12:44:46.037 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.827+0000 7fa3912ac640 10 client.4429.objecter ms_dispatch 0x564ada7af9d0 osd_map(26..26 src has 1..26) v4 2024-05-15T12:44:46.041 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:44:46.041 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-05-15T12:44:46.041 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-05-15T12:44:46.041 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-05-15T12:44:46.042 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-05-15T12:44:46.042 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-05-15T12:44:46.042 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-05-15T12:44:46.042 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:44:46.042 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-05-15T12:44:46.042 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:44:46.042 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-05-15T12:44:46.042 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2024-05-15T12:44:46.042 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2024-05-15T12:44:46.042 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_rados_api_service.log 2024-05-15T12:44:46.042 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-05-15T12:44:46.042 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ api_c_write_operations: /' 2024-05-15T12:44:46.042 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:44:46.042 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:44:46.043 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:44:46.043 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:46.043 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:44:46.043 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:46.043 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:44:46.043 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-05-15T12:44:46.043 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:46.043 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:46.043 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:44:46.043 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.043 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:44:46.043 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.043 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:44:46.043 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:44:46.043 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:44:46.044 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:44:46.044 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:44:46.044 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.044 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo open_pools_parallel 2024-05-15T12:44:46.044 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.044 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:44:46.044 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.044 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:44:46.044 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:44:46.044 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:44:46.044 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.044 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:44:46.044 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:44:46.044 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-05-15T12:44:46.044 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.044 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:44:46.045 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.045 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:44:46.045 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-05-15T12:44:46.045 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ api_service: /' 2024-05-15T12:44:46.045 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=open_pools_parallel 2024-05-15T12:44:46.045 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40574 2024-05-15T12:44:46.045 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test open_pools_parallel on pid 40574' 2024-05-15T12:44:46.045 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.827+0000 7fa3912ac640 test open_pools_parallel on pid 40574 2024-05-15T12:44:46.049 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=40574 2024-05-15T12:44:46.049 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-15T12:44:46.049 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:44:46.049 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-05-15T12:44:46.049 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.049 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:44:46.049 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.049 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:44:46.049 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:46.049 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:44:46.050 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:44:46.050 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:44:46.050 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.050 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:44:46.050 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.050 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:44:46.050 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:44:46.050 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:46.050 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s delete_pools_parallel 2024-05-15T12:44:46.050 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' delete_pools_parallel' 2024-05-15T12:44:46.050 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:44:46.050 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:44:46.050 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:44:46.050 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2024-05-15T12:44:46.051 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.051 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:44:46.051 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.051 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:44:46.051 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:44:46.051 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:46.051 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo delete_pools_parallel 2024-05-15T12:44:46.051 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-05-15T12:44:46.051 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:44:46.051 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-05-15T12:44:46.051 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=delete_pools_parallel 2024-05-15T12:44:46.051 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-05-15T12:44:46.051 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:44:46.051 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.051 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40591 2024-05-15T12:44:46.052 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test delete_pools_parallel on pid 40591' 2024-05-15T12:44:46.052 INFO:tasks.workunit.client.0.smithi117.stdout:test delete_pools_parallel on pid 40591 2024-05-15T12:44:46.064 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=40591 2024-05-15T12:44:46.064 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/40470/exe 2024-05-15T12:44:46.065 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-05-15T12:44:46.065 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-05-15T12:44:46.065 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:44:46.065 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:44:46.065 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:44:46.065 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:44:46.065 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.065 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:44:46.065 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.065 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:44:46.065 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:44:46.065 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.065 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:44:46.065 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.065 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:44:46.066 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.066 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:44:46.066 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.066 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:44:46.066 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-05-15T12:44:46.066 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-05-15T12:44:46.066 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s cls 2024-05-15T12:44:46.066 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' cls' 2024-05-15T12:44:46.066 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-05-15T12:44:46.066 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-05-15T12:44:46.066 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-05-15T12:44:46.066 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-05-15T12:44:46.066 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-05-15T12:44:46.066 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-05-15T12:44:46.066 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:44:46.067 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-05-15T12:44:46.067 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:44:46.067 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-05-15T12:44:46.067 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2024-05-15T12:44:46.067 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ api_c_read_operations: /' 2024-05-15T12:44:46.067 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:44:46.067 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:44:46.067 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:44:46.067 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.067 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:44:46.067 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.067 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:44:46.067 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:44:46.067 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.067 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:44:46.068 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.068 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:44:46.068 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.068 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:44:46.068 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.068 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:44:46.068 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-05-15T12:44:46.068 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-05-15T12:44:46.068 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2024-05-15T12:44:46.068 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:46.068 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:44:46.068 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:46.068 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:44:46.068 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:44:46.068 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:44:46.069 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo cls 2024-05-15T12:44:46.069 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-05-15T12:44:46.069 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:44:46.069 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-05-15T12:44:46.069 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:44:46.069 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-05-15T12:44:46.069 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.069 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-05-15T12:44:46.069 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:44:46.069 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.069 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:44:46.069 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:44:46.069 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.069 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:44:46.069 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.070 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:44:46.070 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.070 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:44:46.070 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:46.070 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-05-15T12:44:46.070 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:46.070 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.070 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:44:46.070 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.070 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:44:46.070 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:46.070 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:44:46.070 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:44:46.070 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:44:46.070 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:46.071 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-05-15T12:44:46.071 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:44:46.071 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:46.071 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:44:46.071 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-05-15T12:44:46.071 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:46.071 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:46.071 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:44:46.071 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.071 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:44:46.071 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.071 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:44:46.071 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:44:46.071 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.071 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:44:46.072 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.072 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:44:46.072 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:44:46.072 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2024-05-15T12:44:46.072 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:44:46.072 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:44:46.072 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:44:46.072 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.072 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-05-15T12:44:46.072 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:44:46.072 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.072 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:44:46.072 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:44:46.072 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-05-15T12:44:46.072 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.073 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:44:46.073 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.073 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:44:46.073 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-05-15T12:44:46.073 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-05-15T12:44:46.073 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.073 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:44:46.073 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.073 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:44:46.073 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:44:46.073 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-05-15T12:44:46.073 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:46.073 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.073 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:44:46.073 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-05-15T12:44:46.074 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.074 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:44:46.074 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:44:46.074 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.074 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:44:46.074 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.074 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:44:46.074 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:46.074 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:46.074 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.074 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:44:46.074 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.074 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:44:46.074 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:46.075 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=cls 2024-05-15T12:44:46.075 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40624 2024-05-15T12:44:46.075 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test cls on pid 40624' 2024-05-15T12:44:46.075 INFO:tasks.workunit.client.0.smithi117.stdout:test cls on pid 40624 2024-05-15T12:44:46.087 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=40624 2024-05-15T12:44:46.087 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-05-15T12:44:46.087 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:44:46.087 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cls 2>&1 | tee ceph_test_neorados_cls.log | sed "s/^/ cls: /"' 2024-05-15T12:44:46.087 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s cmd 2024-05-15T12:44:46.087 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' cmd' 2024-05-15T12:44:46.088 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:44:46.088 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-05-15T12:44:46.088 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-05-15T12:44:46.088 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:44:46.088 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.088 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-05-15T12:44:46.088 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:44:46.088 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:44:46.088 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:44:46.088 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-05-15T12:44:46.088 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.088 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:44:46.088 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:44:46.088 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.088 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:44:46.089 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-05-15T12:44:46.089 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:44:46.089 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-05-15T12:44:46.089 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.089 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:44:46.089 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.089 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:44:46.089 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.089 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-05-15T12:44:46.089 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-05-15T12:44:46.089 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.089 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:44:46.089 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.089 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:44:46.089 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:46.089 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:44:46.090 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.090 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:44:46.090 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:44:46.090 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.090 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:44:46.090 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.090 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:44:46.090 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:46.090 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:46.090 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.090 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:44:46.090 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.090 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:44:46.090 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:46.090 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:46.091 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:44:46.091 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:46.091 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:44:46.091 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:44:46.091 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:44:46.091 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:44:46.091 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:44:46.091 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:44:46.091 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:46.091 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:44:46.091 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:46.091 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:44:46.091 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-05-15T12:44:46.091 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:46.091 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:46.092 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:44:46.092 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.092 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:44:46.092 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.092 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:44:46.092 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:44:46.092 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.092 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:44:46.092 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.092 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:44:46.092 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:44:46.092 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:44:46.092 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:44:46.092 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/40517/exe 2024-05-15T12:44:46.092 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:44:46.093 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.093 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:44:46.093 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.093 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:44:46.093 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:44:46.093 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:46.093 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-05-15T12:44:46.093 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:44:46.093 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:44:46.093 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo cmd 2024-05-15T12:44:46.093 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:44:46.093 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-05-15T12:44:46.093 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-05-15T12:44:46.093 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:44:46.093 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:44:46.093 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-05-15T12:44:46.094 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-05-15T12:44:46.094 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.094 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:44:46.094 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-05-15T12:44:46.094 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-05-15T12:44:46.094 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.094 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:44:46.094 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:44:46.094 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:44:46.094 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-05-15T12:44:46.094 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.094 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:44:46.094 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:44:46.094 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-05-15T12:44:46.094 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.095 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:44:46.095 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:44:46.095 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-05-15T12:44:46.095 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-05-15T12:44:46.095 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.095 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-05-15T12:44:46.095 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.095 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:44:46.095 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.095 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:44:46.095 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:46.095 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:44:46.095 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.095 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:44:46.095 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:44:46.095 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cls_remote_reads.xml 2024-05-15T12:44:46.096 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:46.096 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ api_cls_remote_reads: /' 2024-05-15T12:44:46.096 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_rados_api_cls_remote_reads.log 2024-05-15T12:44:46.096 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/40545/exe 2024-05-15T12:44:46.096 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-05-15T12:44:46.096 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:44:46.096 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:44:46.096 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:44:46.096 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.096 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:44:46.096 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.096 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:44:46.096 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:44:46.096 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:46.096 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-05-15T12:44:46.097 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-05-15T12:44:46.097 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-05-15T12:44:46.097 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-05-15T12:44:46.097 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-05-15T12:44:46.097 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-05-15T12:44:46.097 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-05-15T12:44:46.097 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:44:46.097 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-05-15T12:44:46.097 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:44:46.097 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-05-15T12:44:46.097 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=cmd 2024-05-15T12:44:46.097 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40671 2024-05-15T12:44:46.097 INFO:tasks.workunit.client.0.smithi117.stdout:test cmd on pid 40671 2024-05-15T12:44:46.105 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test cmd on pid 40671' 2024-05-15T12:44:46.105 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=40671 2024-05-15T12:44:46.105 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-05-15T12:44:46.105 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:44:46.105 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s handler_error 2024-05-15T12:44:46.106 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ list_parallel: /' 2024-05-15T12:44:46.106 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:44:46.106 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cmd 2>&1 | tee ceph_test_neorados_cmd.log | sed "s/^/ cmd: /"' 2024-05-15T12:44:46.106 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:44:46.106 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:44:46.106 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.106 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:44:46.106 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.106 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:44:46.106 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:44:46.106 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.106 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:44:46.106 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.106 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:44:46.107 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_rados_list_parallel.log 2024-05-15T12:44:46.107 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.107 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:44:46.107 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.107 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:44:46.107 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-05-15T12:44:46.107 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:44:46.107 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:44:46.107 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:44:46.107 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.107 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:44:46.107 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.107 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:44:46.107 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:44:46.107 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.108 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:44:46.108 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.108 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:44:46.108 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.108 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:44:46.108 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.108 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:44:46.108 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-05-15T12:44:46.108 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-05-15T12:44:46.108 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2024-05-15T12:44:46.108 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:44:46.108 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-05-15T12:44:46.108 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-05-15T12:44:46.108 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:44:46.108 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.109 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-05-15T12:44:46.109 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-05-15T12:44:46.109 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' handler_error' 2024-05-15T12:44:46.109 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:44:46.109 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:44:46.109 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:44:46.109 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-05-15T12:44:46.109 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.109 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:44:46.109 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.109 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:44:46.109 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:44:46.109 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.109 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:44:46.109 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.110 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:44:46.110 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:44:46.110 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-05-15T12:44:46.110 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.110 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.110 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:44:46.110 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.110 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:44:46.110 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo handler_error 2024-05-15T12:44:46.110 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:44:46.110 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.110 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:44:46.110 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-05-15T12:44:46.110 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:44:46.111 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.111 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:44:46.111 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.111 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:44:46.111 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:46.111 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:46.111 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.111 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:44:46.111 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.111 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:44:46.111 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:46.111 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-05-15T12:44:46.111 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-05-15T12:44:46.111 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:46.111 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:44:46.111 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:46.112 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:44:46.112 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:44:46.112 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:46.112 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:44:46.112 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:46.112 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=handler_error 2024-05-15T12:44:46.112 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:44:46.112 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:44:46.112 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40689 2024-05-15T12:44:46.112 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test handler_error on pid 40689' 2024-05-15T12:44:46.113 INFO:tasks.workunit.client.0.smithi117.stdout:test handler_error on pid 40689 2024-05-15T12:44:46.132 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=40689 2024-05-15T12:44:46.132 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-05-15T12:44:46.132 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:44:46.132 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:44:46.132 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:44:46.132 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:44:46.132 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.132 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:44:46.132 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.132 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:44:46.132 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:44:46.132 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_handler_error 2>&1 | tee ceph_test_neorados_handler_error.log | sed "s/^/ handler_error: /"' 2024-05-15T12:44:46.132 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-05-15T12:44:46.133 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.133 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:44:46.133 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.133 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:44:46.133 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-05-15T12:44:46.133 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-05-15T12:44:46.133 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.133 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:44:46.133 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:46.133 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:44:46.133 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:46.133 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:44:46.133 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:44:46.133 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.134 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:44:46.134 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:44:46.134 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:46.134 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:44:46.134 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:44:46.134 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-05-15T12:44:46.134 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-05-15T12:44:46.134 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:44:46.134 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.134 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-05-15T12:44:46.134 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-05-15T12:44:46.134 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:44:46.134 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-05-15T12:44:46.134 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.134 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:44:46.135 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.135 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:44:46.135 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:44:46.135 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.135 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:44:46.135 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:44:46.135 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.135 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:44:46.135 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:44:46.135 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:44:46.135 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:46.135 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:46.135 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.135 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:46.136 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:44:46.136 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.136 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:44:46.136 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:44:46.136 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:46.136 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:46.136 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:44:46.136 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-05-15T12:44:46.136 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:46.136 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:46.136 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:44:46.136 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.136 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:44:46.136 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:44:46.136 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:44:46.137 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:44:46.137 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:46.137 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:44:46.137 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:46.137 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:44:46.137 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-05-15T12:44:46.137 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:46.137 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.137 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:44:46.137 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:46.137 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:44:46.137 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.137 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:44:46.137 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.137 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:44:46.138 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:44:46.138 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.138 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:44:46.138 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.138 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:44:46.138 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:44:46.138 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:44:46.138 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.138 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:44:46.138 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.138 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:44:46.138 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:44:46.138 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s io 2024-05-15T12:44:46.138 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' io' 2024-05-15T12:44:46.138 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/40591/exe 2024-05-15T12:44:46.139 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/40624/exe 2024-05-15T12:44:46.139 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:44:46.139 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-05-15T12:44:46.139 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-05-15T12:44:46.139 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:44:46.139 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:44:46.139 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:44:46.139 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.139 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:44:46.139 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.139 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:44:46.139 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:44:46.139 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-05-15T12:44:46.139 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.139 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:44:46.139 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.140 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:44:46.140 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-05-15T12:44:46.140 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-05-15T12:44:46.140 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.140 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:44:46.140 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.140 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:44:46.140 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:46.140 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-05-15T12:44:46.140 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-05-15T12:44:46.140 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-05-15T12:44:46.140 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-05-15T12:44:46.140 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-05-15T12:44:46.140 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-05-15T12:44:46.140 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:44:46.141 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-05-15T12:44:46.141 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:44:46.141 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-05-15T12:44:46.141 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:44:46.141 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:44:46.141 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:44:46.141 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:46.141 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:44:46.141 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:46.141 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:44:46.141 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-05-15T12:44:46.141 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2024-05-15T12:44:46.141 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:46.141 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:46.141 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:44:46.142 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.142 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:44:46.142 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ delete_pools_parallel: /' 2024-05-15T12:44:46.142 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.142 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:44:46.142 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-05-15T12:44:46.142 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-05-15T12:44:46.142 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-05-15T12:44:46.142 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-05-15T12:44:46.142 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-05-15T12:44:46.142 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-05-15T12:44:46.142 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:44:46.142 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:44:46.142 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-05-15T12:44:46.143 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:44:46.143 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.143 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:44:46.143 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.143 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:44:46.143 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-05-15T12:44:46.143 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:44:46.143 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/40574/exe 2024-05-15T12:44:46.143 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_neorados_cls 2024-05-15T12:44:46.143 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2024-05-15T12:44:46.143 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_neorados_cls.log 2024-05-15T12:44:46.143 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-05-15T12:44:46.143 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:44:46.143 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:44:46.143 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:44:46.144 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.144 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:44:46.144 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.144 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:44:46.144 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:44:46.144 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:46.144 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo io 2024-05-15T12:44:46.144 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-05-15T12:44:46.144 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-05-15T12:44:46.144 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-05-15T12:44:46.144 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-05-15T12:44:46.144 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-05-15T12:44:46.144 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-05-15T12:44:46.144 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-05-15T12:44:46.144 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:44:46.144 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-05-15T12:44:46.145 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:44:46.145 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-05-15T12:44:46.145 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:44:46.145 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:44:46.145 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:44:46.145 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.145 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:44:46.145 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.145 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:44:46.145 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:44:46.145 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:46.145 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2024-05-15T12:44:46.145 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:44:46.145 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:44:46.145 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:44:46.146 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.146 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:44:46.146 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.146 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:44:46.146 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:44:46.146 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.146 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:44:46.146 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.146 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:44:46.146 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.146 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:44:46.146 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.146 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:44:46.146 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-05-15T12:44:46.146 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ cls: /' 2024-05-15T12:44:46.147 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-05-15T12:44:46.147 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:46.147 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:44:46.147 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:46.147 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:44:46.147 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:44:46.147 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:44:46.147 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=io 2024-05-15T12:44:46.147 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40766 2024-05-15T12:44:46.147 INFO:tasks.workunit.client.0.smithi117.stdout:test io on pid 40766 2024-05-15T12:44:46.155 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test io on pid 40766' 2024-05-15T12:44:46.155 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=40766 2024-05-15T12:44:46.155 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-05-15T12:44:46.155 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:44:46.155 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:44:46.155 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:44:46.155 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:44:46.155 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:46.156 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:44:46.156 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:46.156 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:44:46.156 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-05-15T12:44:46.156 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:46.156 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:46.156 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:44:46.156 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.156 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:44:46.156 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.156 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:44:46.156 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:44:46.156 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.156 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:44:46.157 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.157 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:44:46.157 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:44:46.157 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:44:46.157 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:44:46.157 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:44:46.157 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.157 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:44:46.157 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.157 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:44:46.157 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:44:46.157 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.157 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:44:46.157 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.157 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:44:46.157 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_io 2>&1 | tee ceph_test_neorados_io.log | sed "s/^/ io: /"' 2024-05-15T12:44:46.158 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.158 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:44:46.158 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.158 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:44:46.158 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-05-15T12:44:46.158 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s ec_io 2024-05-15T12:44:46.158 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' ec_io' 2024-05-15T12:44:46.158 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo ec_io 2024-05-15T12:44:46.158 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2024-05-15T12:44:46.158 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/40671/exe 2024-05-15T12:44:46.158 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ open_pools_parallel: /' 2024-05-15T12:44:46.158 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-05-15T12:44:46.158 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:44:46.158 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-05-15T12:44:46.159 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-05-15T12:44:46.159 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:44:46.159 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.159 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-05-15T12:44:46.159 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-05-15T12:44:46.159 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-05-15T12:44:46.159 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-05-15T12:44:46.159 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-05-15T12:44:46.159 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-05-15T12:44:46.159 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-05-15T12:44:46.159 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-05-15T12:44:46.159 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:44:46.159 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-05-15T12:44:46.159 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:44:46.159 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.160 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:44:46.160 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.160 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:44:46.160 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:44:46.160 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-05-15T12:44:46.160 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:44:46.160 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-05-15T12:44:46.160 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.160 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:44:46.160 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.160 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:44:46.160 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:46.160 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:46.160 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.160 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:44:46.161 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.161 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:44:46.161 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:46.161 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_neorados_cmd 2024-05-15T12:44:46.161 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ cmd: /' 2024-05-15T12:44:46.161 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_neorados_cmd.log 2024-05-15T12:44:46.161 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-05-15T12:44:46.161 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-05-15T12:44:46.161 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=ec_io 2024-05-15T12:44:46.161 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40793 2024-05-15T12:44:46.161 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test ec_io on pid 40793' 2024-05-15T12:44:46.161 INFO:tasks.workunit.client.0.smithi117.stdout:test ec_io on pid 40793 2024-05-15T12:44:46.163 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=40793 2024-05-15T12:44:46.163 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-05-15T12:44:46.163 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:44:46.163 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s list 2024-05-15T12:44:46.163 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' list' 2024-05-15T12:44:46.163 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo list 2024-05-15T12:44:46.163 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-05-15T12:44:46.163 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=list 2024-05-15T12:44:46.163 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40807 2024-05-15T12:44:46.163 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test list on pid 40807' 2024-05-15T12:44:46.163 INFO:tasks.workunit.client.0.smithi117.stdout:test list on pid 40807 2024-05-15T12:44:46.170 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=40807 2024-05-15T12:44:46.170 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-05-15T12:44:46.170 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:44:46.171 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:46.171 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:44:46.171 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:46.171 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:44:46.171 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:44:46.171 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_io 2>&1 | tee ceph_test_neorados_ec_io.log | sed "s/^/ ec_io: /"' 2024-05-15T12:44:46.171 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:44:46.171 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:44:46.171 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:44:46.171 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.171 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:44:46.171 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.171 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:44:46.171 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:44:46.171 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-05-15T12:44:46.171 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.172 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:44:46.172 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.172 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:44:46.172 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-05-15T12:44:46.172 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-05-15T12:44:46.172 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.172 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:44:46.172 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.172 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:44:46.172 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:46.172 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:44:46.172 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-05-15T12:44:46.172 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-05-15T12:44:46.172 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:44:46.172 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.173 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-05-15T12:44:46.173 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_list 2>&1 | tee ceph_test_neorados_list.log | sed "s/^/ list: /"' 2024-05-15T12:44:46.173 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s ec_list 2024-05-15T12:44:46.173 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' ec_list' 2024-05-15T12:44:46.173 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo ec_list 2024-05-15T12:44:46.173 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:44:46.173 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:44:46.173 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:44:46.173 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:44:46.173 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-05-15T12:44:46.173 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-05-15T12:44:46.173 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:46.173 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:44:46.173 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:46.174 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:44:46.174 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-05-15T12:44:46.174 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:46.174 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:44:46.174 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-05-15T12:44:46.174 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:46.174 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:44:46.174 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.174 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:44:46.174 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.174 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:44:46.174 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.174 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:44:46.175 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:44:46.175 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.175 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.175 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:44:46.175 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:44:46.175 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.175 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:44:46.175 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:44:46.175 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:44:46.175 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:44:46.175 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.175 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-05-15T12:44:46.175 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:44:46.175 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.175 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:44:46.176 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:46.176 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:46.176 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.176 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:44:46.176 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.176 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:44:46.176 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:46.176 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-05-15T12:44:46.176 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:44:46.176 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.176 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-05-15T12:44:46.176 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=ec_list 2024-05-15T12:44:46.176 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40840 2024-05-15T12:44:46.176 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test ec_list on pid 40840' 2024-05-15T12:44:46.177 INFO:tasks.workunit.client.0.smithi117.stdout:test ec_list on pid 40840 2024-05-15T12:44:46.185 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=40840 2024-05-15T12:44:46.185 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-05-15T12:44:46.185 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:44:46.185 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-05-15T12:44:46.186 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:44:46.186 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-05-15T12:44:46.186 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.186 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:44:46.186 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.186 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:44:46.186 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:44:46.186 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.186 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:44:46.186 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.186 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:44:46.186 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:46.186 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:46.186 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.186 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:44:46.187 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.187 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:44:46.187 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:46.187 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/40689/exe 2024-05-15T12:44:46.187 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-05-15T12:44:46.187 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-05-15T12:44:46.187 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-05-15T12:44:46.187 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-05-15T12:44:46.187 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-05-15T12:44:46.187 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-05-15T12:44:46.187 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-05-15T12:44:46.187 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:44:46.187 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-05-15T12:44:46.187 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_list 2>&1 | tee ceph_test_neorados_ec_list.log | sed "s/^/ ec_list: /"' 2024-05-15T12:44:46.187 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:44:46.187 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-05-15T12:44:46.188 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:44:46.188 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-05-15T12:44:46.188 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-05-15T12:44:46.188 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:44:46.188 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.188 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-05-15T12:44:46.188 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_neorados_handler_error 2024-05-15T12:44:46.188 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s misc 2024-05-15T12:44:46.188 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' misc' 2024-05-15T12:44:46.188 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo misc 2024-05-15T12:44:46.188 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:44:46.188 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_neorados_handler_error.log 2024-05-15T12:44:46.188 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:44:46.188 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:44:46.188 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.188 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:44:46.189 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.189 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:44:46.189 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:44:46.189 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-05-15T12:44:46.189 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.189 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:44:46.189 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.189 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:44:46.189 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-05-15T12:44:46.189 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-05-15T12:44:46.189 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.189 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:44:46.189 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.189 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:44:46.189 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:46.190 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ handler_error: /' 2024-05-15T12:44:46.190 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-05-15T12:44:46.190 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:44:46.190 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-05-15T12:44:46.190 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.190 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:44:46.190 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.190 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:44:46.190 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:44:46.190 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.190 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:44:46.190 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.190 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:44:46.190 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:46.190 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:46.191 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.191 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:44:46.191 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.191 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:44:46.191 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:46.191 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-05-15T12:44:46.191 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:44:46.191 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:44:46.191 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:44:46.191 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.191 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:44:46.191 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.191 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:44:46.191 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:44:46.191 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:46.192 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=misc 2024-05-15T12:44:46.192 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40872 2024-05-15T12:44:46.192 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test misc on pid 40872' 2024-05-15T12:44:46.192 INFO:tasks.workunit.client.0.smithi117.stdout:test misc on pid 40872 2024-05-15T12:44:46.195 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=40872 2024-05-15T12:44:46.195 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-05-15T12:44:46.195 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:44:46.195 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s pool 2024-05-15T12:44:46.195 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' pool' 2024-05-15T12:44:46.195 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:44:46.195 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:44:46.195 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:44:46.195 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.195 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:44:46.196 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.196 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:44:46.196 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:44:46.196 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-05-15T12:44:46.196 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.196 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:44:46.196 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.196 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:44:46.196 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-05-15T12:44:46.196 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-05-15T12:44:46.196 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.196 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:44:46.196 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.196 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:44:46.196 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo pool 2024-05-15T12:44:46.197 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:46.197 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-05-15T12:44:46.197 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=pool 2024-05-15T12:44:46.197 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40894 2024-05-15T12:44:46.197 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test pool on pid 40894' 2024-05-15T12:44:46.197 INFO:tasks.workunit.client.0.smithi117.stdout:test pool on pid 40894 2024-05-15T12:44:46.212 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=40894 2024-05-15T12:44:46.212 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-05-15T12:44:46.212 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:44:46.212 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:44:46.212 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:44:46.212 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:44:46.212 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.212 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:44:46.212 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.212 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:44:46.212 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:44:46.212 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-05-15T12:44:46.212 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.213 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:44:46.213 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.213 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:44:46.213 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-05-15T12:44:46.213 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-05-15T12:44:46.213 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.213 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:44:46.213 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.213 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:44:46.213 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:46.213 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:44:46.213 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:44:46.213 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:44:46.213 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.213 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:44:46.214 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.214 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:44:46.214 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:44:46.214 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.214 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:44:46.214 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.214 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:44:46.214 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.214 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:44:46.214 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.214 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:44:46.214 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-05-15T12:44:46.214 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:44:46.214 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:44:46.215 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:44:46.215 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.215 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_pool 2>&1 | tee ceph_test_neorados_pool.log | sed "s/^/ pool: /"' 2024-05-15T12:44:46.215 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:44:46.215 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.215 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:44:46.215 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:44:46.215 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:46.215 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_misc 2>&1 | tee ceph_test_neorados_misc.log | sed "s/^/ misc: /"' 2024-05-15T12:44:46.215 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:44:46.215 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-05-15T12:44:46.215 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:44:46.215 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-05-15T12:44:46.215 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-05-15T12:44:46.215 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-05-15T12:44:46.216 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:44:46.216 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.216 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-05-15T12:44:46.216 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:44:46.216 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.216 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-05-15T12:44:46.216 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:44:46.216 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:44:46.216 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-05-15T12:44:46.216 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:44:46.216 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:44:46.216 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.216 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:44:46.216 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.216 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:44:46.216 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:44:46.217 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:46.217 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-05-15T12:44:46.217 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-05-15T12:44:46.217 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:44:46.217 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:44:46.217 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:44:46.217 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.217 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:44:46.217 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.217 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:44:46.217 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:44:46.217 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:46.217 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s read_operations 2024-05-15T12:44:46.217 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.217 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:44:46.218 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:44:46.218 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.218 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:44:46.218 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-05-15T12:44:46.218 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' read_operations' 2024-05-15T12:44:46.218 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:44:46.218 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.218 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.218 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:44:46.218 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.218 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:44:46.218 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:44:46.218 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.218 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:44:46.219 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.219 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:44:46.219 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.219 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:44:46.219 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:46.219 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:44:46.219 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-05-15T12:44:46.219 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:46.219 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:46.219 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.219 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:44:46.219 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.219 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:44:46.219 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:46.220 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:46.220 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.220 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:44:46.220 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.220 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:44:46.221 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:46.222 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo read_operations 2024-05-15T12:44:46.222 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:46.222 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:44:46.222 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:46.222 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:44:46.222 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:44:46.222 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-05-15T12:44:46.222 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:44:46.222 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:44:46.222 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:44:46.222 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.222 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:44:46.222 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.222 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:44:46.223 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:44:46.223 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.223 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:44:46.223 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.223 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:44:46.223 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.223 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:44:46.223 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.223 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:44:46.223 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-05-15T12:44:46.223 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:44:46.223 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:44:46.223 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:44:46.223 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:44:46.223 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.223 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:44:46.224 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.224 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:44:46.224 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-05-15T12:44:46.224 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:44:46.224 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.224 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:44:46.224 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.224 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:44:46.224 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:44:46.224 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:44:46.224 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:44:46.224 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.224 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:44:46.224 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.224 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:44:46.225 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:44:46.225 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.225 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:44:46.225 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.225 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:44:46.225 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.225 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.225 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:44:46.225 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.225 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:44:46.225 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-05-15T12:44:46.225 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:44:46.225 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.225 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:44:46.225 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-05-15T12:44:46.225 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=read_operations 2024-05-15T12:44:46.225 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-05-15T12:44:46.226 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40952 2024-05-15T12:44:46.226 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:44:46.226 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test read_operations on pid 40952' 2024-05-15T12:44:46.226 INFO:tasks.workunit.client.0.smithi117.stdout:test read_operations on pid 40952 2024-05-15T12:44:46.240 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=40952 2024-05-15T12:44:46.240 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-05-15T12:44:46.240 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:44:46.240 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:44:46.240 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:44:46.240 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:46.240 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:44:46.240 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_read_operations 2>&1 | tee ceph_test_neorados_read_operations.log | sed "s/^/ read_operations: /"' 2024-05-15T12:44:46.240 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s snapshots 2024-05-15T12:44:46.240 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:46.240 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:44:46.240 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' snapshots' 2024-05-15T12:44:46.240 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-05-15T12:44:46.241 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:46.241 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:46.241 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:44:46.241 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:46.241 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:44:46.241 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:46.241 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-05-15T12:44:46.241 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:44:46.241 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:44:46.241 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.241 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:44:46.241 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.241 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:44:46.241 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:44:46.242 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.242 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:44:46.242 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.242 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:44:46.242 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:44:46.242 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:44:46.242 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:44:46.242 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-05-15T12:44:46.242 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/40766/exe 2024-05-15T12:44:46.242 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-05-15T12:44:46.242 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:44:46.242 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.242 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-05-15T12:44:46.242 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-05-15T12:44:46.242 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-05-15T12:44:46.243 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:46.243 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:44:46.243 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:44:46.243 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-05-15T12:44:46.243 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:44:46.243 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:46.243 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:44:46.243 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.243 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:44:46.243 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:44:46.243 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.243 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:44:46.243 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:44:46.243 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:44:46.243 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:44:46.243 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-05-15T12:44:46.244 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:44:46.244 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.244 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:44:46.244 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:44:46.244 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:44:46.244 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.244 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:44:46.244 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.244 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:44:46.244 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.244 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:44:46.244 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:46.244 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:44:46.244 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:44:46.245 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-05-15T12:44:46.245 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:46.245 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:46.245 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.245 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:46.245 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:44:46.245 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:44:46.245 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.245 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.245 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-05-15T12:44:46.245 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:44:46.245 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:44:46.245 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.245 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:44:46.245 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:46.245 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-05-15T12:44:46.246 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:44:46.246 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-05-15T12:44:46.246 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.246 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:44:46.246 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:46.246 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.246 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:44:46.246 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:46.246 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-05-15T12:44:46.246 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:46.246 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:44:46.246 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.246 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:44:46.246 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.246 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:44:46.247 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:44:46.247 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-05-15T12:44:46.247 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-05-15T12:44:46.247 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-05-15T12:44:46.247 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-05-15T12:44:46.247 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.247 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:44:46.247 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:44:46.247 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.247 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-05-15T12:44:46.247 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:44:46.247 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-05-15T12:44:46.247 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:44:46.247 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:44:46.248 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo snapshots 2024-05-15T12:44:46.248 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ io: /' 2024-05-15T12:44:46.248 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/40840/exe 2024-05-15T12:44:46.248 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:44:46.248 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:44:46.248 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:44:46.248 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:46.248 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:44:46.248 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:46.248 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:44:46.248 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-05-15T12:44:46.248 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:46.248 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:46.248 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:44:46.248 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-05-15T12:44:46.248 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.249 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:44:46.249 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.249 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:44:46.249 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:44:46.249 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:44:46.249 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.249 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:44:46.249 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:44:46.249 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.249 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:44:46.249 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:44:46.249 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:44:46.249 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.249 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:44:46.249 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.250 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:44:46.250 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:44:46.250 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-05-15T12:44:46.250 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.250 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:44:46.250 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.250 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:44:46.250 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-05-15T12:44:46.250 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_neorados_io 2024-05-15T12:44:46.250 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-05-15T12:44:46.250 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.250 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:44:46.250 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.250 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:44:46.250 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:46.251 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/40807/exe 2024-05-15T12:44:46.251 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-05-15T12:44:46.251 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:46.251 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:44:46.251 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:46.251 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:44:46.251 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:44:46.251 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-05-15T12:44:46.251 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-05-15T12:44:46.251 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-05-15T12:44:46.251 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-05-15T12:44:46.251 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-05-15T12:44:46.251 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-05-15T12:44:46.251 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-05-15T12:44:46.251 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:44:46.252 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-05-15T12:44:46.252 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:44:46.252 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-05-15T12:44:46.252 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-05-15T12:44:46.252 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-05-15T12:44:46.252 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-05-15T12:44:46.252 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:44:46.252 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-05-15T12:44:46.252 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-05-15T12:44:46.252 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_neorados_io.log 2024-05-15T12:44:46.252 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-05-15T12:44:46.252 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_neorados_ec_list 2024-05-15T12:44:46.252 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:44:46.252 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-05-15T12:44:46.252 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_neorados_ec_list.log 2024-05-15T12:44:46.253 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:44:46.253 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-05-15T12:44:46.253 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ ec_list: /' 2024-05-15T12:44:46.253 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=snapshots 2024-05-15T12:44:46.253 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:44:46.253 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:44:46.253 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:44:46.253 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.253 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:44:46.253 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.253 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40993 2024-05-15T12:44:46.253 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:44:46.254 INFO:tasks.workunit.client.0.smithi117.stdout:test snapshots on pid 40993 2024-05-15T12:44:46.262 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test snapshots on pid 40993' 2024-05-15T12:44:46.262 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=40993 2024-05-15T12:44:46.262 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-05-15T12:44:46.263 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:44:46.263 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:44:46.263 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-05-15T12:44:46.263 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.263 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:44:46.263 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.263 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:44:46.263 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-05-15T12:44:46.263 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-05-15T12:44:46.263 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.263 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:44:46.263 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.263 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:44:46.263 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:46.263 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_neorados_list.log 2024-05-15T12:44:46.263 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ list: /' 2024-05-15T12:44:46.264 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:44:46.264 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:44:46.264 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:44:46.264 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:46.264 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:44:46.264 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:46.264 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:44:46.264 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-05-15T12:44:46.264 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:46.264 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:46.264 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:44:46.264 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.264 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:44:46.264 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.264 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:44:46.264 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:44:46.265 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.265 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:44:46.265 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.265 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:44:46.265 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:44:46.265 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_neorados_list 2024-05-15T12:44:46.265 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:44:46.265 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:44:46.265 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:44:46.265 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.265 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:44:46.265 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.265 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:44:46.265 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:44:46.265 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:46.265 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_snapshots 2>&1 | tee ceph_test_neorados_snapshots.log | sed "s/^/ snapshots: /"' 2024-05-15T12:44:46.265 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s watch_notify 2024-05-15T12:44:46.266 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:44:46.266 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:44:46.266 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:44:46.266 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.266 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:44:46.266 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.266 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:44:46.266 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' watch_notify' 2024-05-15T12:44:46.266 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:44:46.266 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:46.266 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:44:46.266 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-05-15T12:44:46.266 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-05-15T12:44:46.266 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:44:46.266 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.267 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-05-15T12:44:46.267 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/40793/exe 2024-05-15T12:44:46.267 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-05-15T12:44:46.267 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:44:46.267 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-05-15T12:44:46.267 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.267 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:44:46.267 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.267 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:44:46.267 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:44:46.267 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.267 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:44:46.267 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-05-15T12:44:46.267 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.267 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:44:46.267 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:46.267 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:46.267 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.268 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:44:46.268 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.268 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:44:46.268 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:46.268 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-05-15T12:44:46.268 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:44:46.268 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:44:46.268 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo watch_notify 2024-05-15T12:44:46.268 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:44:46.268 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.268 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:44:46.268 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.268 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:44:46.268 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:44:46.268 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:44:46.268 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.269 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:44:46.269 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.269 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:44:46.269 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.269 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:44:46.269 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.269 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:44:46.269 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-05-15T12:44:46.269 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:44:46.269 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:44:46.269 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:44:46.269 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:44:46.269 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.269 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:44:46.269 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.270 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:44:46.270 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:44:46.270 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=watch_notify 2024-05-15T12:44:46.270 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:46.270 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-05-15T12:44:46.270 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-05-15T12:44:46.270 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=41014 2024-05-15T12:44:46.270 INFO:tasks.workunit.client.0.smithi117.stdout:test watch_notify on pid 41014 2024-05-15T12:44:46.288 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test watch_notify on pid 41014' 2024-05-15T12:44:46.288 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=41014 2024-05-15T12:44:46.288 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-05-15T12:44:46.288 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-05-15T12:44:46.288 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-05-15T12:44:46.288 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:44:46.288 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-05-15T12:44:46.288 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-05-15T12:44:46.288 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:44:46.288 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-05-15T12:44:46.288 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:44:46.289 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-05-15T12:44:46.289 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_watch_notify 2>&1 | tee ceph_test_neorados_watch_notify.log | sed "s/^/ watch_notify: /"' 2024-05-15T12:44:46.289 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-05-15T12:44:46.289 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_neorados_ec_io.log 2024-05-15T12:44:46.289 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:44:46.289 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.289 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:44:46.289 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-05-15T12:44:46.289 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:44:46.289 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.289 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:44:46.289 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-05-15T12:44:46.289 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:44:46.289 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.289 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-05-15T12:44:46.290 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:44:46.290 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.290 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:44:46.290 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.290 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:44:46.290 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.290 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:44:46.290 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.290 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:44:46.290 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-05-15T12:44:46.290 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-05-15T12:44:46.290 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:46.290 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:44:46.290 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:46.291 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:44:46.291 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:44:46.291 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-05-15T12:44:46.291 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:44:46.291 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-05-15T12:44:46.291 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.291 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:44:46.291 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.291 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:44:46.291 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s write_operations 2024-05-15T12:44:46.291 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:44:46.291 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.291 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' write_operations' 2024-05-15T12:44:46.291 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:44:46.291 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.292 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:44:46.292 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:46.292 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:46.292 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.292 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:44:46.292 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.292 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:44:46.292 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:46.292 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_neorados_ec_io 2024-05-15T12:44:46.292 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:44:46.292 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ ec_io: /' 2024-05-15T12:44:46.292 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:44:46.292 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:44:46.292 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:44:46.292 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.293 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:44:46.293 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.293 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:44:46.293 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:44:46.293 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-05-15T12:44:46.293 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.293 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:44:46.293 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:46.293 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:44:46.293 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:46.293 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:44:46.293 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:44:46.293 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.293 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:44:46.293 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-05-15T12:44:46.293 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-05-15T12:44:46.294 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.294 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:44:46.294 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.294 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:44:46.294 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:46.294 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:44:46.294 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:44:46.294 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:44:46.294 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:44:46.294 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:46.294 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:44:46.294 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:46.294 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:44:46.294 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-05-15T12:44:46.294 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:46.295 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:46.295 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:44:46.295 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.295 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:44:46.295 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.295 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:44:46.295 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:44:46.295 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.295 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:44:46.295 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.295 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:44:46.295 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:44:46.295 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:44:46.295 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:44:46.295 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:44:46.296 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.296 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:44:46.296 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.296 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:44:46.296 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:44:46.296 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-05-15T12:44:46.296 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.296 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:44:46.296 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.296 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:44:46.296 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-05-15T12:44:46.296 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-05-15T12:44:46.296 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.296 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:44:46.296 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.297 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:44:46.297 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:46.297 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:44:46.297 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:44:46.297 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:44:46.297 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:46.297 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:44:46.297 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:46.297 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:44:46.297 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/40894/exe 2024-05-15T12:44:46.297 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-05-15T12:44:46.297 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:46.297 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:46.297 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:44:46.297 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.297 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:44:46.298 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.298 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:44:46.298 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:44:46.298 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.298 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:44:46.298 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.298 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:44:46.298 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:44:46.298 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/40872/exe 2024-05-15T12:44:46.298 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-05-15T12:44:46.298 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-05-15T12:44:46.298 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:44:46.298 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:44:46.298 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:44:46.298 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.299 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-05-15T12:44:46.299 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-05-15T12:44:46.299 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-05-15T12:44:46.299 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-05-15T12:44:46.299 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-05-15T12:44:46.299 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-05-15T12:44:46.299 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:44:46.299 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-05-15T12:44:46.299 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-05-15T12:44:46.299 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-05-15T12:44:46.299 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-05-15T12:44:46.299 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-05-15T12:44:46.299 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-05-15T12:44:46.299 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:44:46.299 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-05-15T12:44:46.300 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:44:46.300 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-05-15T12:44:46.300 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:44:46.300 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-05-15T12:44:46.300 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-05-15T12:44:46.300 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo write_operations 2024-05-15T12:44:46.300 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-05-15T12:44:46.300 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:44:46.300 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.300 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:44:46.300 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:44:46.300 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:46.300 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_neorados_misc.log 2024-05-15T12:44:46.300 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_neorados_misc 2024-05-15T12:44:46.300 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:44:46.301 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:44:46.301 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:44:46.301 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.301 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:44:46.301 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.301 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:44:46.301 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:44:46.301 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:46.301 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:44:46.301 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:44:46.301 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:44:46.301 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.301 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:44:46.301 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.301 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:44:46.302 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:44:46.302 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.302 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:44:46.302 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.302 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:44:46.302 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.302 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:44:46.302 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.302 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:44:46.302 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-05-15T12:44:46.302 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=write_operations 2024-05-15T12:44:46.302 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=41060 2024-05-15T12:44:46.302 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test write_operations on pid 41060' 2024-05-15T12:44:46.303 INFO:tasks.workunit.client.0.smithi117.stdout:test write_operations on pid 41060 2024-05-15T12:44:46.325 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=41060 2024-05-15T12:44:46.325 INFO:tasks.workunit.client.0.smithi117.stderr:+ ret=0 2024-05-15T12:44:46.326 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:44:46.326 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-05-15T12:44:46.326 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40021 2024-05-15T12:44:46.326 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 40021 2024-05-15T12:44:46.326 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_write_operations 2>&1 | tee ceph_test_neorados_write_operations.log | sed "s/^/ write_operations: /"' 2024-05-15T12:44:46.326 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:44:46.326 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ misc: /' 2024-05-15T12:44:46.326 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_neorados_pool.log 2024-05-15T12:44:46.326 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-05-15T12:44:46.326 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ pool: /' 2024-05-15T12:44:46.326 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-05-15T12:44:46.326 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:44:46.326 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:44:46.326 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:44:46.327 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:44:46.327 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.327 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:44:46.327 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.327 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:44:46.327 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.327 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:44:46.327 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.327 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:44:46.327 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.327 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:44:46.327 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-05-15T12:44:46.327 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.327 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:44:46.327 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.328 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:44:46.328 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-05-15T12:44:46.328 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_neorados_pool 2024-05-15T12:44:46.328 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-05-15T12:44:46.328 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-05-15T12:44:46.328 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:44:46.328 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-05-15T12:44:46.328 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.328 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:44:46.328 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.328 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:44:46.328 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:44:46.328 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:44:46.328 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.328 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:44:46.329 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.329 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:44:46.329 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:44:46.329 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:44:46.329 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:46.329 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:44:46.329 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.329 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:44:46.329 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.329 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.329 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:44:46.329 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:44:46.329 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.329 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:44:46.329 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:46.329 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:44:46.330 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.330 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:44:46.330 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.330 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:44:46.330 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.330 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:44:46.330 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.330 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:44:46.330 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-05-15T12:44:46.330 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-05-15T12:44:46.330 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-05-15T12:44:46.330 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:46.330 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:44:46.330 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:46.331 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:44:46.331 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:44:46.331 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:44:46.331 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:46.331 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:44:46.331 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:46.331 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:44:46.331 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:44:46.331 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:44:46.331 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:44:46.331 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:44:46.331 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:44:46.331 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:46.331 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:44:46.331 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:46.331 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:44:46.332 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-05-15T12:44:46.332 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:46.332 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:44:46.332 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:46.332 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:44:46.332 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:44:46.332 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.332 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:44:46.332 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:44:46.332 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.332 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:44:46.332 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.332 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:44:46.332 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.332 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:44:46.332 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:44:46.333 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.333 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:44:46.333 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.333 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:44:46.333 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:44:46.333 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-05-15T12:44:46.333 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.333 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:44:46.333 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.333 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:44:46.333 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:44:46.333 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-05-15T12:44:46.333 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-05-15T12:44:46.333 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.333 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:44:46.334 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.334 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:44:46.334 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:46.334 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/41014/exe 2024-05-15T12:44:46.334 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:46.334 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:44:46.334 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:44:46.334 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:46.334 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:44:46.334 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-05-15T12:44:46.334 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:44:46.334 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:44:46.334 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:44:46.334 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:46.335 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:44:46.335 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:46.335 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:44:46.335 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-05-15T12:44:46.335 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:46.335 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:46.335 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:44:46.335 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:44:46.335 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.335 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:44:46.335 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.335 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:44:46.335 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:44:46.335 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.335 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:44:46.336 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.336 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:44:46.336 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-05-15T12:44:46.336 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-05-15T12:44:46.336 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-05-15T12:44:46.336 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:44:46.336 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-05-15T12:44:46.336 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-05-15T12:44:46.336 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-05-15T12:44:46.336 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:44:46.336 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-05-15T12:44:46.336 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:44:46.336 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-05-15T12:44:46.336 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/40952/exe 2024-05-15T12:44:46.336 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:44:46.337 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:44:46.337 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:44:46.337 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ watch_notify: /' 2024-05-15T12:44:46.337 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:46.337 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:44:46.337 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_neorados_watch_notify 2024-05-15T12:44:46.337 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:46.337 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:44:46.337 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-05-15T12:44:46.337 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:46.337 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:46.337 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:44:46.337 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.337 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:44:46.337 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.338 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:44:46.338 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:44:46.338 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.338 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:44:46.338 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.338 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:44:46.338 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:44:46.338 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-05-15T12:44:46.338 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-05-15T12:44:46.338 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-05-15T12:44:46.338 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-05-15T12:44:46.338 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-05-15T12:44:46.338 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-05-15T12:44:46.338 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-05-15T12:44:46.338 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:44:46.339 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-05-15T12:44:46.339 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:44:46.339 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-05-15T12:44:46.339 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_neorados_read_operations 2024-05-15T12:44:46.339 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ read_operations: /' 2024-05-15T12:44:46.339 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:44:46.339 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_neorados_watch_notify.log 2024-05-15T12:44:46.339 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:44:46.339 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:44:46.339 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.339 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:44:46.339 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.339 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:44:46.339 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:44:46.339 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_neorados_read_operations.log 2024-05-15T12:44:46.339 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:44:46.340 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/40993/exe 2024-05-15T12:44:46.340 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-05-15T12:44:46.340 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-05-15T12:44:46.340 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-05-15T12:44:46.340 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-05-15T12:44:46.340 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-05-15T12:44:46.340 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-05-15T12:44:46.340 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-05-15T12:44:46.340 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:44:46.340 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-05-15T12:44:46.340 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:44:46.340 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-05-15T12:44:46.340 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:44:46.340 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:44:46.341 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:44:46.341 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.341 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:44:46.341 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ snapshots: /' 2024-05-15T12:44:46.341 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.341 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:44:46.341 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:44:46.341 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.341 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:44:46.341 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.341 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:44:46.341 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_neorados_snapshots.log 2024-05-15T12:44:46.341 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.341 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:44:46.341 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.342 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:44:46.342 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-05-15T12:44:46.342 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_neorados_snapshots 2024-05-15T12:44:46.342 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-05-15T12:44:46.342 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:46.342 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:44:46.342 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:46.342 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:44:46.342 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:44:46.342 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:44:46.342 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:44:46.342 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:44:46.342 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:44:46.342 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:44:46.342 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:44:46.343 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:46.343 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:44:46.343 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-05-15T12:44:46.343 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-05-15T12:44:46.343 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:44:46.343 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:44:46.343 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.343 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:44:46.343 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.343 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:44:46.343 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:44:46.343 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:44:46.343 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:44:46.343 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-05-15T12:44:46.343 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:44:46.344 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:44:46.344 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/41060/exe 2024-05-15T12:44:46.344 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-05-15T12:44:46.344 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-05-15T12:44:46.344 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-05-15T12:44:46.344 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-05-15T12:44:46.344 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-05-15T12:44:46.344 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-05-15T12:44:46.344 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-05-15T12:44:46.344 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:44:46.344 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-05-15T12:44:46.344 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:44:46.344 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-05-15T12:44:46.344 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ write_operations: /' 2024-05-15T12:44:46.344 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_neorados_write_operations.log 2024-05-15T12:44:46.345 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_neorados_write_operations 2024-05-15T12:44:47.209 INFO:tasks.workunit.client.0.smithi117.stdout: 3 client.4429.objecter handle_osd_map got epochs [26,26] > 0 2024-05-15T12:44:47.210 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.827+0000 7fa3912ac640 3 client.4429.objecter handle_osd_map decoding full epoch 26 2024-05-15T12:44:47.210 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.827+0000 7fa3912ac640 20 client.4429.objecter dump_active .. 0 homeless 2024-05-15T12:44:47.210 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.827+0000 7fa39684b780 10 librados: wait_for_osdmap done waiting 2024-05-15T12:44:47.210 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.827+0000 7fa39684b780 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:44:47.210 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.827+0000 7fa39684b780 1 -- 172.21.15.117:0/4241893909 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa388008890 con 0x564ada8d4330 2024-05-15T12:44:47.210 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.855+0000 7fa3912ac640 1 -- 172.21.15.117:0/4241893909 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa388059130 con 0x564ada8d4330 2024-05-15T12:44:47.210 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.855+0000 7fa3912ac640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-05-15T12:44:47.210 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.855+0000 7fa3922ae640 10 client.4429.objecter _get_latest_version latest 26, have it 2024-05-15T12:44:47.210 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.855+0000 7fa39684b780 10 client.4429.objecter create_pool name=ceph_test_rados_api_asio 2024-05-15T12:44:47.210 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.855+0000 7fa39684b780 10 client.4429.objecter pool_op_submit 1 2024-05-15T12:44:47.210 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.855+0000 7fa39684b780 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:44:47.210 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:45.855+0000 7fa39684b780 1 -- 172.21.15.117:0/4241893909 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- pool_op(create pool 0 tid 1 name ceph_test_rados_api_asio v0) v4 -- 0x564ada7bc420 con 0x564ada8d4330 2024-05-15T12:44:47.210 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:47.188+0000 7fa3912ac640 1 -- 172.21.15.117:0/4241893909 <== mon.1 v2:172.21.15.178:3300/0 7 ==== pool_op_reply(tid 1 (0) Success v27) v1 ==== 43+0+0 (secure 0 0 0) 0x7fa3880133e0 con 0x564ada8d4330 2024-05-15T12:44:47.211 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:47.188+0000 7fa3912ac640 10 client.4429.objecter ms_dispatch 0x564ada7af9d0 pool_op_reply(tid 1 (0) Success v27) v1 2024-05-15T12:44:47.211 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:47.188+0000 7fa3912ac640 10 client.4429.objecter handle_pool_op_reply pool_op_reply(tid 1 (0) Success v27) v1 2024-05-15T12:44:47.211 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:47.188+0000 7fa3912ac640 10 client.4429.objecter have request 1 at 0x564ada8d1820 Op: create 2024-05-15T12:44:47.211 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:47.188+0000 7fa3912ac640 20 client.4429.objecter waiting for client to reach epoch 27 before calling back 2024-05-15T12:44:47.211 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:47.188+0000 7fa3912ac640 10 client.4429.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:44:47.211 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:47.188+0000 7fa3912ac640 10 monclient: _renew_subs 2024-05-15T12:44:47.211 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:47.188+0000 7fa3912ac640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:44:47.211 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:47.188+0000 7fa3912ac640 1 -- 172.21.15.117:0/4241893909 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=27}) v3 -- 0x7fa380070e70 con 0x564ada8d4330 2024-05-15T12:44:47.211 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:47.188+0000 7fa3912ac640 10 client.4429.objecter done 2024-05-15T12:44:47.211 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:47.219+0000 7fa3912ac640 1 -- 172.21.15.117:0/4241893909 <== mon.1 v2:172.21.15.178:3300/0 8 ==== osd_map(27..27 src has 1..27) v4 ==== 2860+0+0 (secure 0 0 0) 0x7fa388057660 con 0x564ada8d4330 2024-05-15T12:44:47.211 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:47.219+0000 7fa3912ac640 10 client.4429.objecter ms_dispatch 0x564ada7af9d0 osd_map(27..27 src has 1..27) v4 2024-05-15T12:44:47.211 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:47.219+0000 7fa3912ac640 3 client.4429.objecter handle_osd_map got epochs [27,27] > 26 2024-05-15T12:44:49.720 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:47.219+0000 7fa3912ac640 3 client.4429.objecter handle_osd_map decoding incremental epoch 27 2024-05-15T12:44:49.720 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:47.219+0000 7fa3912ac640 20 client.4429.objecter dump_active .. 0 homeless 2024-05-15T12:44:49.720 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:47.219+0000 7fa39684b780 10 librados: set snap read head -> 1 2024-05-15T12:44:49.720 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:47.219+0000 7fa39684b780 10 librados: writefull oid=exist nspace= 2024-05-15T12:44:49.720 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:47.219+0000 7fa39684b780 10 client.4429.objecter _op_submit op 0x564ada976330 2024-05-15T12:44:49.720 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:47.219+0000 7fa39684b780 20 client.4429.objecter _calc_target epoch 27 base exist @6 precalc_pgid 0 pgid 0.0 is_write 2024-05-15T12:44:49.721 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:47.219+0000 7fa39684b780 20 client.4429.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-05-15T12:44:49.721 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:47.219+0000 7fa39684b780 10 client.4429.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [6,3] primary 6 2024-05-15T12:44:49.721 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:47.219+0000 7fa39684b780 1 --2- 172.21.15.117:0/4241893909 >> [v2:172.21.15.178:6800/1383036222,v1:172.21.15.178:6801/1383036222] conn(0x564ada976cf0 0x564ada8b0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:44:49.721 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:47.219+0000 7fa39684b780 20 client.4429.objecter _get_session s=0x564ada8d10d0 osd=6 3 2024-05-15T12:44:49.721 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:47.219+0000 7fa39684b780 10 client.4429.objecter _op_submit oid exist '@6' '@6' [writefull 0~5 in=5b] tid 2 osd.6 2024-05-15T12:44:49.721 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:47.219+0000 7fa39684b780 20 client.4429.objecter get_session s=0x564ada8d10d0 osd=6 3 2024-05-15T12:44:49.721 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:47.219+0000 7fa39684b780 15 client.4429.objecter _session_op_assign 6 2 2024-05-15T12:44:49.721 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:47.219+0000 7fa39684b780 15 client.4429.objecter _send_op 2 to 6.1 on osd.6 2024-05-15T12:44:49.721 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:47.219+0000 7fa39684b780 1 -- 172.21.15.117:0/4241893909 --> [v2:172.21.15.178:6800/1383036222,v1:172.21.15.178:6801/1383036222] -- osd_op(unknown.0.0:2 6.1 6:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) v9 -- 0x564ada8b0dc0 con 0x564ada976cf0 2024-05-15T12:44:49.721 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:47.219+0000 7fa39684b780 20 client.4429.objecter put_session s=0x564ada8d10d0 osd=6 4 2024-05-15T12:44:49.721 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:47.219+0000 7fa39684b780 5 client.4429.objecter 1 in flight 2024-05-15T12:44:49.721 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:47.220+0000 7fa3942b2640 1 --2- 172.21.15.117:0/4241893909 >> [v2:172.21.15.178:6800/1383036222,v1:172.21.15.178:6801/1383036222] conn(0x564ada976cf0 0x564ada8b0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:44:49.721 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:47.220+0000 7fa3942b2640 10 monclient: get_auth_request con 0x564ada976cf0 auth_method 0 2024-05-15T12:44:49.721 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:47.223+0000 7fa3912ac640 10 client.4429.objecter ms_handle_connect 0x564ada976cf0 2024-05-15T12:44:49.721 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:47.223+0000 7fa3942b2640 1 --2- 172.21.15.117:0/4241893909 >> [v2:172.21.15.178:6800/1383036222,v1:172.21.15.178:6801/1383036222] conn(0x564ada976cf0 0x564ada8b0880 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:44:49.722 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:48.730+0000 7fa37bfff640 10 monclient: tick 2024-05-15T12:44:49.722 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:48.730+0000 7fa37bfff640 10 monclient: _check_auth_tickets 2024-05-15T12:44:49.722 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:48.730+0000 7fa37bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:44:49.722 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:48.730+0000 7fa37bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:44:49.722 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:49.730+0000 7fa37bfff640 10 monclient: tick 2024-05-15T12:44:57.202 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:49.730+0000 7fa37bfff640 api_service: [==========] Running 4 tests from 1 test suite. 2024-05-15T12:44:57.203 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: [----------] Global test environment set-up. 2024-05-15T12:44:57.203 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: [----------] 4 tests from LibRadosService 2024-05-15T12:44:57.203 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2024-05-15T12:44:57.203 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5320 ms) 2024-05-15T12:44:57.203 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2024-05-15T12:44:57.203 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: [ OK ] LibRadosService.RegisterLate (68 ms) 2024-05-15T12:44:57.203 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2024-05-15T12:44:57.203 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: cluster: 2024-05-15T12:44:57.203 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: id: 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:44:57.203 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: health: HEALTH_WARN 2024-05-15T12:44:57.203 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 29 pool(s) do not have an application enabled 2024-05-15T12:44:57.203 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 2024-05-15T12:44:57.203 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: services: 2024-05-15T12:44:57.203 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2024-05-15T12:44:57.203 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-05-15T12:44:57.204 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: osd: 8 osds: 8 up (since 111s), 8 in (since 2m) 2024-05-15T12:44:57.204 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-05-15T12:44:57.204 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 2024-05-15T12:44:57.204 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: data: 2024-05-15T12:44:57.204 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: pools: 37 pools, 1129 pgs 2024-05-15T12:44:57.204 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: objects: 2 objects, 449 KiB 2024-05-15T12:44:57.204 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2024-05-15T12:44:57.204 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: pgs: 97.166% pgs unknown 2024-05-15T12:44:57.204 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 2.037% pgs not active 2024-05-15T12:44:57.204 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 1097 unknown 2024-05-15T12:44:57.204 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 23 creating+peering 2024-05-15T12:44:57.204 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 9 active+clean 2024-05-15T12:44:57.204 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 2024-05-15T12:44:57.204 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 2024-05-15T12:44:57.204 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: cluster: 2024-05-15T12:44:57.204 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: id: 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:44:57.205 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: health: HEALTH_WARN 2024-05-15T12:44:57.205 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 15 pool(s) do not have an application enabled 2024-05-15T12:44:57.205 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 2024-05-15T12:44:57.205 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: services: 2024-05-15T12:44:57.205 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2024-05-15T12:44:57.205 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-05-15T12:44:57.205 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: osd: 8 osds: 8 up (since 114s), 8 in (since 2m) 2024-05-15T12:44:57.205 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 2024-05-15T12:44:57.205 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: data: 2024-05-15T12:44:57.205 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: pools: 35 pools, 1017 pgs 2024-05-15T12:44:57.205 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: objects: 3 objects, 449 KiB 2024-05-15T12:44:57.205 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2024-05-15T12:44:57.205 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: pgs: 71.976% pgs unknown 2024-05-15T12:44:57.205 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 26.844% pgs not active 2024-05-15T12:44:57.205 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 732 unknown 2024-05-15T12:44:57.205 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 243 creating+peering 2024-05-15T12:44:57.206 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 30 creating+activating 2024-05-15T12:44:57.206 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 12 active+clean 2024-05-15T12:44:57.206 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 2024-05-15T12:44:57.206 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: io: 2024-05-15T12:44:57.206 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: client: 209 B/s wr, 0 op/s rd, 0 op/s wr 2024-05-15T12:44:57.206 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 2024-05-15T12:44:57.206 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 2024-05-15T12:44:57.206 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: cluster: 2024-05-15T12:44:57.206 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: id: 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T12:44:57.206 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: health: HEALTH_WARN 2024-05-15T12:44:57.206 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 19 pool(s) do not have an application enabled 2024-05-15T12:44:57.206 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 2024-05-15T12:44:57.206 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: services: 2024-05-15T12:44:57.206 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2024-05-15T12:44:57.206 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-05-15T12:44:57.207 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: osd: 8 osds: 8 up (since 116s), 8 in (since 2m) 2024-05-15T12:44:57.207 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2024-05-15T12:44:57.207 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-05-15T12:44:57.207 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 2024-05-15T12:44:57.207 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: data: 2024-05-15T12:44:57.207 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: pools: 38 pools, 1089 pgs 2024-05-15T12:44:57.207 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: objects: 3 objects, 449 KiB 2024-05-15T12:44:59.863 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB 10 monclient: _check_auth_tickets 2024-05-15T12:44:59.863 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:49.730+0000 7fa37bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:44:59.864 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:50.730+0000 7fa37bfff640 10 monclient: tick 2024-05-15T12:44:59.864 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:50.730+0000 7fa37bfff640 10 monclient: _check_auth_tickets 2024-05-15T12:44:59.864 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:50.730+0000 7fa37bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:44:59.864 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:50.766+0000 7fa395d15640 10 client.4429.objecter tick 2024-05-15T12:44:59.864 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:51.730+0000 7fa37bfff640 10 monclient: tick 2024-05-15T12:44:59.864 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:51.730+0000 7fa37bfff640 10 monclient: _check_auth_tickets 2024-05-15T12:44:59.864 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:51.730+0000 7fa37bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:44:59.864 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:52.730+0000 7fa37bfff640 10 monclient: tick 2024-05-15T12:44:59.864 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:52.730+0000 7fa37bfff640 10 monclient: _check_auth_tickets 2024-05-15T12:44:59.864 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:52.730+0000 7fa37bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:44:59.864 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:53.731+0000 7fa37bfff640 10 monclient: tick 2024-05-15T12:44:59.864 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:53.731+0000 7fa37bfff640 10 monclient: _check_auth_tickets 2024-05-15T12:44:59.864 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:53.731+0000 7fa37bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:44:59.864 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:54.731+0000 7fa37bfff640 10 monclient: tick 2024-05-15T12:44:59.865 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:54.731+0000 7fa37bfff640 10 monclient: _check_auth_tickets 2024-05-15T12:44:59.865 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:54.731+0000 7fa37bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:44:59.865 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:55.731+0000 7fa37bfff640 10 monclient: tick 2024-05-15T12:44:59.865 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:55.731+0000 7fa37bfff640 10 monclient: _check_auth_tickets 2024-05-15T12:44:59.865 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:55.731+0000 7fa37bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:44:59.865 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:55.767+0000 7fa395d15640 10 client.4429.objecter tick 2024-05-15T12:44:59.865 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:56.731+0000 7fa37bfff640 10 monclient: tick 2024-05-15T12:44:59.865 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:56.731+0000 7fa37bfff640 10 monclient: _check_auth_tickets 2024-05-15T12:44:59.865 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:56.731+0000 7fa37bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:44:59.865 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:57.731+0000 7fa37bfff640 10 monclient: tick 2024-05-15T12:44:59.865 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:57.731+0000 7fa37bfff640 10 monclient: _check_auth_tickets 2024-05-15T12:44:59.865 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:57.731+0000 7fa37bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:44:59.865 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:58.731+0000 7fa37bfff640 10 monclient: tick 2024-05-15T12:44:59.865 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:58.731+0000 7fa37bfff640 10 monclient: _check_auth_tickets 2024-05-15T12:44:59.866 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:58.731+0000 7fa37bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:44:59.866 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:58.731+0000 7fa37bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:44:59.866 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:59.731+0000 7fa37bfff640 10 monclient: tick 2024-05-15T12:44:59.866 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:59.731+0000 7fa37bfff640 10 monclient: _check_auth_tickets 2024-05-15T12:44:59.866 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:59.731+0000 7fa37bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:44:59.866 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:59.874+0000 7fa3912ac640 1 -- 172.21.15.117:0/4241893909 <== osd.6 v2:172.21.15.178:6800/1383036222 1 ==== osd_map(28..31 src has 1..31) v4 ==== 24485+0+0 (crc 0 0 0) 0x7fa38c004b40 con 0x564ada976cf0 2024-05-15T12:45:00.761 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:59.874+0000 7fa3912ac640 10 client.4429.objecter ms_dispatch 0x564ada7af9d0 osd_map(28..31 src has 1..31) v4 2024-05-15T12:45:00.761 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:59.874+0000 7fa3912ac640 3 client.4429.objecter handle_osd_map got epochs [28,31] > 27 2024-05-15T12:45:00.761 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:59.874+0000 7fa3912ac640 3 client.4429.objecter handle_osd_map decoding incremental epoch 28 2024-05-15T12:45:00.761 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:59.874+0000 7fa3912ac640 10 client.4429.objecter checking op 2 2024-05-15T12:45:00.761 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:59.874+0000 7fa3912ac640 20 client.4429.objecter _calc_target epoch 28 base exist @6 precalc_pgid 0 pgid 0.0 is_write 2024-05-15T12:45:00.761 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:59.874+0000 7fa3912ac640 20 client.4429.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-05-15T12:45:00.761 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:59.874+0000 7fa3912ac640 3 client.4429.objecter handle_osd_map decoding incremental epoch 29 2024-05-15T12:45:00.761 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:59.874+0000 7fa3912ac640 10 client.4429.objecter checking op 2 2024-05-15T12:45:00.761 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:59.874+0000 7fa3912ac640 20 client.4429.objecter _calc_target epoch 29 base exist @6 precalc_pgid 0 pgid 0.0 is_write 2024-05-15T12:45:00.761 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:59.874+0000 7fa3912ac640 20 client.4429.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-05-15T12:45:00.761 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:59.874+0000 7fa3912ac640 3 client.4429.objecter handle_osd_map decoding incremental epoch 30 2024-05-15T12:45:00.762 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:59.874+0000 7fa3912ac640 10 client.4429.objecter checking op 2 2024-05-15T12:45:00.762 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:59.874+0000 7fa3912ac640 20 client.4429.objecter _calc_target epoch 30 base exist @6 precalc_pgid 0 pgid 0.0 is_write 2024-05-15T12:45:00.762 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:59.874+0000 7fa3912ac640 20 client.4429.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-05-15T12:45:00.762 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:59.874+0000 7fa3912ac640 3 client.4429.objecter handle_osd_map decoding incremental epoch 31 2024-05-15T12:45:00.762 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:59.874+0000 7fa3912ac640 10 client.4429.objecter checking op 2 2024-05-15T12:45:00.762 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:59.874+0000 7fa3912ac640 20 client.4429.objecter _calc_target epoch 31 base exist @6 precalc_pgid 0 pgid 0.0 is_write 2024-05-15T12:45:00.762 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:59.874+0000 7fa3912ac640 20 client.4429.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-05-15T12:45:00.762 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:59.874+0000 7fa3912ac640 20 client.4429.objecter dump_active .. 0 homeless 2024-05-15T12:45:00.762 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:44:59.874+0000 7fa3912ac640 20 client.4429.objecter 2 6.d3a59201 osd.6 exist [writefull 0~5 in=5b] 2024-05-15T12:45:00.762 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:00.732+0000 7fa37bfff640 10 monclient: tick 2024-05-15T12:45:00.762 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:00.732+0000 7fa37bfff640 10 monclient: _check_auth_tickets 2024-05-15T12:45:00.762 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:00.732+0000 7fa37bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:45:00.762 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:00.768+0000 7fa395d15640 10 client.4429.objecter tick 2024-05-15T12:45:00.762 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:00.768+0000 7fa395d15640 2 client.4429.objecter tid 2 on osd.6 is laggy 2024-05-15T12:45:00.763 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:00.768+0000 7fa395d15640 10 client.4429.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:45:00.763 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:00.768+0000 7fa395d15640 10 monclient: _renew_subs 2024-05-15T12:45:00.763 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:00.768+0000 7fa395d15640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:45:00.763 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:00.768+0000 7fa395d15640 1 -- 172.21.15.117:0/4241893909 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=32}) v3 -- 0x7fa364002f90 con 0x564ada8d4330 2024-05-15T12:45:00.763 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:00.768+0000 7fa395d15640 1 -- 172.21.15.117:0/4241893909 --> [v2:172.21.15.178:6800/1383036222,v1:172.21.15.178:6801/1383036222] -- ping magic: 0 v1 -- 0x7fa364003360 con 0x564ada976cf0 2024-05-15T12:45:03.722 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:00.771+0000 7fa3912ac640 1 -- 172.21.15.117:0/4241893909 <== mon.1 v2:172.21.15.178:3300/0 9 ==== osd_map(32..36 src has 1..36) v4 ==== 7271+0+0 (secure 0 0 0) 0x7fa3880592f0 con 0x564ada8d4330 2024-05-15T12:45:03.722 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:00.771+0000 7fa3912ac640 10 client.4429.objecter ms_dispatch 0x564ada7af9d0 osd_map(32..36 src has 1..36) v4 2024-05-15T12:45:03.722 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:00.771+0000 7fa3912ac640 3 client.4429.objecter handle_osd_map got epochs [32,36] > 31 2024-05-15T12:45:03.722 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:00.771+0000 7fa3912ac640 3 client.4429.objecter handle_osd_map decoding incremental epoch 32 2024-05-15T12:45:03.722 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:00.772+0000 7fa3912ac640 10 client.4429.objecter checking op 2 2024-05-15T12:45:03.722 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:00.772+0000 7fa3912ac640 20 client.4429.objecter _calc_target epoch 32 base exist @6 precalc_pgid 0 pgid 0.0 is_write 2024-05-15T12:45:03.722 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:00.772+0000 7fa3912ac640 20 client.4429.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-05-15T12:45:03.722 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:00.772+0000 7fa3912ac640 3 client.4429.objecter handle_osd_map decoding incremental epoch 33 2024-05-15T12:45:03.723 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:00.772+0000 7fa3912ac640 10 client.4429.objecter checking op 2 2024-05-15T12:45:03.723 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:00.772+0000 7fa3912ac640 20 client.4429.objecter _calc_target epoch 33 base exist @6 precalc_pgid 0 pgid 0.0 is_write 2024-05-15T12:45:03.723 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:00.772+0000 7fa3912ac640 20 client.4429.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-05-15T12:45:03.723 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:00.772+0000 7fa3912ac640 3 client.4429.objecter handle_osd_map decoding incremental epoch 34 2024-05-15T12:45:03.723 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:00.772+0000 7fa3912ac640 10 client.4429.objecter checking op 2 2024-05-15T12:45:03.723 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:00.772+0000 7fa3912ac640 20 client.4429.objecter _calc_target epoch 34 base exist @6 precalc_pgid 0 pgid 0.0 is_write 2024-05-15T12:45:03.723 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:00.772+0000 7fa3912ac640 20 client.4429.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-05-15T12:45:03.723 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:00.772+0000 7fa3912ac640 3 client.4429.objecter handle_osd_map decoding incremental epoch 35 2024-05-15T12:45:03.723 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:00.772+0000 7fa3912ac640 10 client.4429.objecter checking op 2 2024-05-15T12:45:03.723 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:00.772+0000 7fa3912ac640 20 client.4429.objecter _calc_target epoch 35 base exist @6 precalc_pgid 0 pgid 0.0 is_write 2024-05-15T12:45:03.723 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:00.772+0000 7fa3912ac640 20 client.4429.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-05-15T12:45:03.723 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:00.772+0000 7fa3912ac640 3 client.4429.objecter handle_osd_map decoding incremental epoch 36 2024-05-15T12:45:03.723 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:00.772+0000 7fa3912ac640 10 client.4429.objecter checking op 2 2024-05-15T12:45:03.723 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:00.772+0000 7fa3912ac640 20 client.4429.objecter _calc_target epoch 36 base exist @6 precalc_pgid 0 pgid 0.0 is_write 2024-05-15T12:45:03.723 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:00.772+0000 7fa3912ac640 20 client.4429.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-05-15T12:45:03.724 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:00.772+0000 7fa3912ac640 20 client.4429.objecter dump_active .. 0 homeless 2024-05-15T12:45:03.724 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:00.772+0000 7fa3912ac640 20 client.4429.objecter 2 6.d3a59201 osd.6 exist [writefull 0~5 in=5b] 2024-05-15T12:45:03.724 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:01.731+0000 7fa37bfff640 10 monclient: tick 2024-05-15T12:45:03.724 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:01.731+0000 7fa37bfff640 10 monclient: _check_auth_tickets 2024-05-15T12:45:03.724 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:01.731+0000 7fa37bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:45:03.724 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:02.731+0000 7fa37bfff640 10 monclient: tick 2024-05-15T12:45:03.724 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:02.731+0000 7fa37bfff640 10 monclient: _check_auth_tickets 2024-05-15T12:45:03.724 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:02.731+0000 7fa37bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:45:05.767 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:03.731+0000 7fa37bfff640 10 monclient: tick 2024-05-15T12:45:05.767 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:03.731+0000 7fa37bfff640 10 monclient: _check_auth_tickets 2024-05-15T12:45:05.767 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:03.731+0000 7fa37bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:45:05.767 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:04.731+0000 7fa37bfff640 10 monclient: tick 2024-05-15T12:45:05.767 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:04.731+0000 7fa37bfff640 10 monclient: _check_auth_tickets 2024-05-15T12:45:05.767 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:04.731+0000 7fa37bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:45:05.767 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:05.732+0000 7fa37bfff640 10 monclient: tick 2024-05-15T12:45:05.768 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:05.732+0000 7fa37bfff640 10 monclient: _check_auth_tickets 2024-05-15T12:45:05.768 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:05.732+0000 7fa37bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:45:05.768 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:05.768+0000 7fa395d15640 10 client.4429.objecter tick 2024-05-15T12:45:05.768 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:05.768+0000 7fa395d15640 2 client.4429.objecter tid 2 on osd.6 is laggy 2024-05-15T12:45:05.768 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:05.768+0000 7fa395d15640 10 client.4429.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:45:05.768 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:05.768+0000 7fa395d15640 10 monclient: _renew_subs 2024-05-15T12:45:05.768 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:05.768+0000 7fa395d15640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T12:45:05.768 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:05.768+0000 7fa395d15640 1 -- 172.21.15.117:0/4241893909 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=37}) v3 -- 0x7fa364002f90 con 0x564ada8d4330 2024-05-15T12:45:05.768 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:05.768+0000 7fa395d15640 1 -- 172.21.15.117:0/4241893909 --> [v2:172.21.15.178:6800/1383036222,v1:172.21.15.178:6801/1383036222] -- ping magic: 0 v1 -- 0x7fa3640035c0 con 0x564ada976cf0 2024-05-15T12:45:05.768 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:05.776+0000 7fa3912ac640 1 -- 172.21.15.117:0/4241893909 <== mon.1 v2:172.21.15.178:3300/0 10 ==== osd_map(37..40 src has 1..40) v4 ==== 5273+0+0 (secure 0 0 0) 0x7fa388059040 con 0x564ada8d4330 2024-05-15T12:45:05.768 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:05.776+0000 7fa3912ac640 10 client.4429.objecter ms_dispatch 0x564ada7af9d0 osd_map(37..40 src has 1..40) v4 2024-05-15T12:45:05.768 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:05.776+0000 7fa3912ac640 3 client.4429.objecter handle_osd_map got epochs [37,40] > 36 2024-05-15T12:45:05.768 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:05.776+0000 7fa3912ac640 3 client.4429.objecter handle_osd_map decoding incremental epoch 37 2024-05-15T12:45:05.768 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:05.776+0000 7fa3912ac640 10 client.4429.objecter checking op 2 2024-05-15T12:45:05.768 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:05.776+0000 7fa3912ac640 20 client.4429.objecter _calc_target epoch 37 base exist @6 precalc_pgid 0 pgid 0.0 is_write 2024-05-15T12:45:05.769 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:05.776+0000 7fa3912ac640 20 client.4429.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-05-15T12:45:05.769 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:05.776+0000 7fa3912ac640 3 client.4429.objecter handle_osd_map decoding incremental epoch 38 2024-05-15T12:45:05.769 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:05.776+0000 7fa3912ac640 10 client.4429.objecter checking op 2 2024-05-15T12:45:05.769 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:05.776+0000 7fa3912ac640 20 client.4429.objecter _calc_target epoch 38 base exist @6 precalc_pgid 0 pgid 0.0 is_write 2024-05-15T12:45:05.769 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:05.776+0000 7fa3912ac640 20 client.4429.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-05-15T12:45:05.769 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:05.777+0000 7fa3912ac640 3 client.4429.objecter handle_osd_map decoding incremental epoch 39 2024-05-15T12:45:05.769 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:05.777+0000 7fa3912ac640 10 client.4429.objecter checking op 2 2024-05-15T12:45:05.769 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:05.777+0000 7fa3912ac640 20 client.4429.objecter _calc_target epoch 39 base exist @6 precalc_pgid 0 pgid 0.0 is_write 2024-05-15T12:45:08.127 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:05.777+0000 7fa3912ac640 20 client.4429.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-05-15T12:45:08.127 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:05.777+0000 7fa3912ac640 3 client.4429.objecter handle_osd_map decoding incremental epoch 40 2024-05-15T12:45:08.127 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:05.777+0000 7fa3912ac640 10 client.4429.objecter checking op 2 2024-05-15T12:45:08.127 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:05.777+0000 7fa3912ac640 20 client.4429.objecter _calc_target epoch 40 base exist @6 precalc_pgid 0 pgid 0.0 is_write 2024-05-15T12:45:08.127 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:05.777+0000 7fa3912ac640 20 client.4429.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-05-15T12:45:08.127 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:05.777+0000 7fa3912ac640 20 client.4429.objecter dump_active .. 0 homeless 2024-05-15T12:45:08.127 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:05.777+0000 7fa3912ac640 20 client.4429.objecter 2 6.d3a59201 osd.6 exist [writefull 0~5 in=5b] 2024-05-15T12:45:08.127 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:06.732+0000 7fa37bfff640 10 monclient: tick 2024-05-15T12:45:08.127 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:06.732+0000 7fa37bfff640 10 monclient: _check_auth_tickets 2024-05-15T12:45:08.127 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:06.732+0000 7fa37bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:45:08.127 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:07.732+0000 7fa37bfff640 10 monclient: tick 2024-05-15T12:45:08.127 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:07.732+0000 7fa37bfff640 10 monclient: _check_auth_tickets 2024-05-15T12:45:08.128 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:07.732+0000 7fa37bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:45:08.128 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:08.136+0000 7fa3942b2640 1 -- 172.21.15.117:0/4241893909 <== osd.6 v2:172.21.15.178:6800/1383036222 2 ==== osd_op_reply(2 exist [writefull 0~5] v31'1 uv1 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7fa38c001820 con 0x564ada976cf0 2024-05-15T12:45:08.128 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:08.136+0000 7fa3942b2640 10 client.4429.objecter ms_dispatch 0x564ada7af9d0 osd_op_reply(2 exist [writefull 0~5] v31'1 uv1 ondisk = 0) v8 2024-05-15T12:45:08.128 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:08.136+0000 7fa3942b2640 10 client.4429.objecter in handle_osd_op_reply 2024-05-15T12:45:08.128 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:08.136+0000 7fa3942b2640 7 client.4429.objecter handle_osd_op_reply 2 ondisk uv 1 in 6.1 attempt 0 2024-05-15T12:45:08.128 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:08.136+0000 7fa3942b2640 10 client.4429.objecter op 0 rval 0 len 0 2024-05-15T12:45:08.128 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:08.136+0000 7fa3942b2640 15 client.4429.objecter handle_osd_op_reply completed tid 2 2024-05-15T12:45:08.128 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:08.136+0000 7fa3942b2640 15 client.4429.objecter _finish_op 2 2024-05-15T12:45:08.128 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:08.136+0000 7fa3942b2640 20 client.4429.objecter put_session s=0x564ada8d10d0 osd=6 4 2024-05-15T12:45:08.128 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:08.136+0000 7fa3942b2640 15 client.4429.objecter _session_op_remove 6 2 2024-05-15T12:45:08.128 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:08.136+0000 7fa3942b2640 5 client.4429.objecter 0 in flight 2024-05-15T12:45:08.128 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:08.136+0000 7fa39684b780 10 librados: Objecter returned from writefull r=0 2024-05-15T12:45:08.128 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2024-05-15T12:45:08.128 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:08.136+0000 7fa39684b780 10 client.4429.objecter _op_submit op 0x564ada976330 2024-05-15T12:45:08.129 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:08.136+0000 7fa39684b780 20 client.4429.objecter _calc_target epoch 40 base exist @6 precalc_pgid 0 pgid 0.0 is_read 2024-05-15T12:45:08.129 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:08.136+0000 7fa39684b780 20 client.4429.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-05-15T12:45:08.129 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:08.136+0000 7fa39684b780 10 client.4429.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [6,3] primary 6 2024-05-15T12:45:08.129 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:08.136+0000 7fa39684b780 20 client.4429.objecter _get_session s=0x564ada8d10d0 osd=6 3 2024-05-15T12:45:08.129 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:08.136+0000 7fa39684b780 10 client.4429.objecter _op_submit oid exist '@6' '@6' [read 0~256] tid 3 osd.6 2024-05-15T12:45:11.722 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:08.136+0000 7fa39684b780 20 client.4429.objecter get_session s=0x564ada8d10d0 osd=6 3 2024-05-15T12:45:11.723 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:08.136+0000 7fa39684b780 15 client.4429.objecter _session_op_assign 6 3 2024-05-15T12:45:11.723 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:08.136+0000 7fa39684b780 15 client.4429.objecter _send_op 3 to 6.1 on osd.6 2024-05-15T12:45:11.723 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:08.136+0000 7fa39684b780 1 -- 172.21.15.117:0/4241893909 --> [v2:172.21.15.178:6800/1383036222,v1:172.21.15.178:6801/1383036222] -- osd_op(unknown.0.0:3 6.1 6:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x564ada8df5c0 con 0x564ada976cf0 2024-05-15T12:45:11.723 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:08.136+0000 7fa39684b780 20 client.4429.objecter put_session s=0x564ada8d10d0 osd=6 4 2024-05-15T12:45:11.723 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:08.136+0000 7fa39684b780 5 client.4429.objecter 1 in flight 2024-05-15T12:45:11.723 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:08.136+0000 7fa39684b780 10 client.4429.objecter _op_submit op 0x564ada8df960 2024-05-15T12:45:11.723 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:08.136+0000 7fa39684b780 20 client.4429.objecter _calc_target epoch 40 base noexist @6 precalc_pgid 0 pgid 0.0 is_read 2024-05-15T12:45:11.723 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:08.136+0000 7fa39684b780 20 client.4429.objecter _calc_target target noexist @6 -> pgid 6.a7bbaa1 2024-05-15T12:45:11.723 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:08.136+0000 7fa39684b780 10 client.4429.objecter _calc_target raw pgid 6.a7bbaa1 -> actual 6.1 acting [6,3] primary 6 2024-05-15T12:45:11.723 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:08.136+0000 7fa39684b780 20 client.4429.objecter _get_session s=0x564ada8d10d0 osd=6 4 2024-05-15T12:45:11.723 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:08.136+0000 7fa39684b780 10 client.4429.objecter _op_submit oid noexist '@6' '@6' [read 0~256] tid 4 osd.6 2024-05-15T12:45:11.723 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:08.136+0000 7fa39684b780 20 client.4429.objecter get_session s=0x564ada8d10d0 osd=6 4 2024-05-15T12:45:11.724 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:08.136+0000 7fa39684b780 15 client.4429.objecter _session_op_assign 6 4 2024-05-15T12:45:11.724 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:08.136+0000 7fa39684b780 15 client.4429.objecter _send_op 4 to 6.1 on osd.6 2024-05-15T12:45:11.724 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:08.136+0000 7fa39684b780 1 -- 172.21.15.117:0/4241893909 --> [v2:172.21.15.178:6800/1383036222,v1:172.21.15.178:6801/1383036222] -- osd_op(unknown.0.0:4 6.1 6:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x564ada8dfe10 con 0x564ada976cf0 2024-05-15T12:45:11.724 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:08.136+0000 7fa39684b780 20 client.4429.objecter put_session s=0x564ada8d10d0 osd=6 5 2024-05-15T12:45:11.724 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:08.136+0000 7fa39684b780 5 client.4429.objecter 2 in flight 2024-05-15T12:45:11.724 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:08.732+0000 7fa37bfff640 10 monclient: tick 2024-05-15T12:45:11.724 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:08.732+0000 7fa37bfff640 10 monclient: _check_auth_tickets 2024-05-15T12:45:11.724 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:08.732+0000 7fa37bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:45:11.724 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:08.732+0000 7fa37bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:45:11.724 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:09.732+0000 7fa37bfff640 10 monclient: tick 2024-05-15T12:45:11.724 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:09.732+0000 7fa37bfff640 10 monclient: _check_auth_tickets 2024-05-15T12:45:11.724 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:09.732+0000 7fa37bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:45:11.724 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:10.732+0000 7fa37bfff640 10 monclient: tick 2024-05-15T12:45:11.724 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:10.732+0000 7fa37bfff640 10 monclient: _check_auth_tickets 2024-05-15T12:45:11.724 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:10.732+0000 7fa37bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:45:11.725 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:10.769+0000 7fa395d15640 10 client.4429.objecter tick 2024-05-15T12:45:11.725 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:11.732+0000 7fa37bfff640 10 monclient: tick 2024-05-15T12:45:11.725 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:11.732+0000 7fa37bfff640 10 monclient: _check_auth_tickets 2024-05-15T12:45:12.854 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:11.732+0000 7fa37bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:45:12.855 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:12.732+0000 7fa37bfff640 10 monclient: tick 2024-05-15T12:45:12.855 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:12.732+0000 7fa37bfff640 10 monclient: _check_auth_tickets 2024-05-15T12:45:12.855 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:12.732+0000 7fa37bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:45:12.855 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:12.864+0000 7fa3942b2640 1 -- 172.21.15.117:0/4241893909 <== osd.6 v2:172.21.15.178:6800/1383036222 3 ==== osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7fa38c001820 con 0x564ada976cf0 2024-05-15T12:45:12.855 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:12.864+0000 7fa3942b2640 10 client.4429.objecter ms_dispatch 0x564ada7af9d0 osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2024-05-15T12:45:12.855 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:12.864+0000 7fa3942b2640 10 client.4429.objecter in handle_osd_op_reply 2024-05-15T12:45:12.855 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:12.864+0000 7fa3942b2640 7 client.4429.objecter handle_osd_op_reply 3 ondisk uv 1 in 6.1 attempt 0 2024-05-15T12:45:12.855 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:12.864+0000 7fa3942b2640 10 client.4429.objecter op 0 rval 0 len 5 2024-05-15T12:45:12.855 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:12.864+0000 7fa3942b2640 15 client.4429.objecter handle_osd_op_reply completed tid 3 2024-05-15T12:45:12.855 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:12.864+0000 7fa3942b2640 15 client.4429.objecter _finish_op 3 2024-05-15T12:45:12.855 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:12.864+0000 7fa3942b2640 20 client.4429.objecter put_session s=0x564ada8d10d0 osd=6 5 2024-05-15T12:45:12.855 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:12.864+0000 7fa3942b2640 15 client.4429.objecter _session_op_remove 6 3 2024-05-15T12:45:12.855 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:12.864+0000 7fa3942b2640 5 client.4429.objecter 1 in flight 2024-05-15T12:45:12.855 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (4728 ms) 2024-05-15T12:45:12.856 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2024-05-15T12:45:12.856 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:12.864+0000 7fa3942b2640 1 -- 172.21.15.117:0/4241893909 <== osd.6 v2:172.21.15.178:6800/1383036222 4 ==== osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7fa38c001820 con 0x564ada976cf0 2024-05-15T12:45:12.856 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:12.864+0000 7fa3942b2640 10 client.4429.objecter ms_dispatch 0x564ada7af9d0 osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-05-15T12:45:12.856 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:12.864+0000 7fa3942b2640 10 client.4429.objecter in handle_osd_op_reply 2024-05-15T12:45:12.856 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:12.864+0000 7fa3942b2640 7 client.4429.objecter handle_osd_op_reply 4 ondisk uv 0 in 6.1 attempt 0 2024-05-15T12:45:12.856 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:12.864+0000 7fa3942b2640 10 client.4429.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-05-15T12:45:12.856 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:12.864+0000 7fa3942b2640 10 client.4429.objecter op 0 rval 0 len 0 2024-05-15T12:45:12.856 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:12.864+0000 7fa3942b2640 15 client.4429.objecter handle_osd_op_reply completed tid 4 2024-05-15T12:45:12.856 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:12.864+0000 7fa3942b2640 15 client.4429.objecter _finish_op 4 2024-05-15T12:45:12.856 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:12.864+0000 7fa3942b2640 20 client.4429.objecter put_session s=0x564ada8d10d0 osd=6 4 2024-05-15T12:45:12.856 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:12.864+0000 7fa3942b2640 15 client.4429.objecter _session_op_remove 6 4 2024-05-15T12:45:12.856 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:12.864+0000 7fa3942b2640 5 client.4429.objecter 0 in flight 2024-05-15T12:45:12.856 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:12.864+0000 7fa39684b780 10 client.4429.objecter _op_submit op 0x564ada8df960 2024-05-15T12:45:12.856 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:12.864+0000 7fa39684b780 20 client.4429.objecter _calc_target epoch 40 base exist @6 precalc_pgid 0 pgid 0.0 is_read 2024-05-15T12:45:12.857 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:12.864+0000 7fa39684b780 20 client.4429.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-05-15T12:45:13.993 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:12.864+0000 7fa39684b780 10 client.4429.o pool: Running main() from gmock_main.cc 2024-05-15T12:45:13.993 INFO:tasks.workunit.client.0.smithi117.stdout: pool: [==========] Running 6 tests from 1 test suite. 2024-05-15T12:45:13.993 INFO:tasks.workunit.client.0.smithi117.stdout: pool: [----------] Global test environment set-up. 2024-05-15T12:45:13.993 INFO:tasks.workunit.client.0.smithi117.stdout: pool: [----------] 6 tests from NeoRadosPools 2024-05-15T12:45:13.994 INFO:tasks.workunit.client.0.smithi117.stdout: pool: [ RUN ] NeoRadosPools.PoolList 2024-05-15T12:45:13.994 INFO:tasks.workunit.client.0.smithi117.stdout: pool: [ OK ] NeoRadosPools.PoolList (4380 ms) 2024-05-15T12:45:13.994 INFO:tasks.workunit.client.0.smithi117.stdout: pool: [ RUN ] NeoRadosPools.PoolLookup 2024-05-15T12:45:13.994 INFO:tasks.workunit.client.0.smithi117.stdout: pool: [ OK ] NeoRadosPools.PoolLookup (4364 ms) 2024-05-15T12:45:13.994 INFO:tasks.workunit.client.0.smithi117.stdout: pool: [ RUN ] NeoRadosPools.PoolLookupOtherInstance 2024-05-15T12:45:13.994 INFO:tasks.workunit.client.0.smithi117.stdout: pool: [ OK ] NeoRadosPools.PoolLookupOtherInstance (2836 ms) 2024-05-15T12:45:13.994 INFO:tasks.workunit.client.0.smithi117.stdout: pool: [ RUN ] NeoRadosPools.PoolDelete 2024-05-15T12:45:13.994 INFO:tasks.workunit.client.0.smithi117.stdout: pool: [ OK ] NeoRadosPools.PoolDelete (5525 ms) 2024-05-15T12:45:13.994 INFO:tasks.workunit.client.0.smithi117.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateDelete 2024-05-15T12:45:13.994 INFO:tasks.workunit.client.0.smithi117.stdout: pool: [ OK ] NeoRadosPools.PoolCreateDelete (6191 ms) 2024-05-15T12:45:13.994 INFO:tasks.workunit.client.0.smithi117.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateWithCrushRule 2024-05-15T12:45:13.994 INFO:tasks.workunit.client.0.smithi117.stdout: pool: [ OK ] NeoRadosPools.PoolCreateWithCrushRule (4633 ms) 2024-05-15T12:45:13.994 INFO:tasks.workunit.client.0.smithi117.stdout: pool: [----------] 6 tests from NeoRadosPools (27929 ms total) 2024-05-15T12:45:13.994 INFO:tasks.workunit.client.0.smithi117.stdout: pool: 2024-05-15T12:45:13.995 INFO:tasks.workunit.client.0.smithi117.stdout: pool: [----------] Global test environment tear-down 2024-05-15T12:45:13.995 INFO:tasks.workunit.client.0.smithi117.stdout: pool: [==========] 6 tests from 1 test suite ran. (27929 ms total) 2024-05-15T12:45:13.995 INFO:tasks.workunit.client.0.smithi117.stdout: pool: [ PASSED ] 6 tests. 2024-05-15T12:45:14.237 INFO:tasks.workunit.client.0.smithi117.stdout:bjecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [6,3] primary 6 2024-05-15T12:45:14.237 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:12.864+0000 7fa39684b780 20 client.4429.objecter _get_session s=0x564ada8d10d0 osd=6 3 2024-05-15T12:45:14.237 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:12.864+0000 7fa39684b780 10 client.4429.objecter _op_submit oid exist '@6' '@6' [read 0~256] tid 5 osd.6 2024-05-15T12:45:14.237 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:12.864+0000 7fa39684b780 20 client.4429.objecter get_session s=0x564ada8d10d0 osd=6 3 2024-05-15T12:45:14.237 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:12.864+0000 7fa39684b780 15 client.4429.objecter _session_op_assign 6 5 2024-05-15T12:45:14.237 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:12.864+0000 7fa39684b780 15 client.4429.objecter _send_op 5 to 6.1 on osd.6 2024-05-15T12:45:14.237 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:12.864+0000 7fa39684b780 1 -- 172.21.15.117:0/4241893909 --> [v2:172.21.15.178:6800/1383036222,v1:172.21.15.178:6801/1383036222] -- osd_op(unknown.0.0:5 6.1 6:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x564ada8e01e0 con 0x564ada976cf0 2024-05-15T12:45:14.237 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:12.864+0000 7fa39684b780 20 client.4429.objecter put_session s=0x564ada8d10d0 osd=6 4 2024-05-15T12:45:14.237 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:12.864+0000 7fa39684b780 5 client.4429.objecter 1 in flight 2024-05-15T12:45:14.238 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:12.864+0000 7fa39684b780 10 client.4429.objecter _op_submit op 0x564ada976330 2024-05-15T12:45:14.238 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:12.864+0000 7fa39684b780 20 client.4429.objecter _calc_target epoch 40 base noexist @6 precalc_pgid 0 pgid 0.0 is_read 2024-05-15T12:45:14.238 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:12.864+0000 7fa39684b780 20 client.4429.objecter _calc_target target noexist @6 -> pgid 6.a7bbaa1 2024-05-15T12:45:14.238 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:12.864+0000 7fa39684b780 10 client.4429.objecter _calc_target raw pgid 6.a7bbaa1 -> actual 6.1 acting [6,3] primary 6 2024-05-15T12:45:14.238 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:12.864+0000 7fa39684b780 20 client.4429.objecter _get_session s=0x564ada8d10d0 osd=6 4 2024-05-15T12:45:14.238 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:12.864+0000 7fa39684b780 10 client.4429.objecter _op_submit oid noexist '@6' '@6' [read 0~256] tid 6 osd.6 2024-05-15T12:45:14.238 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:12.864+0000 7fa39684b780 20 client.4429.objecter get_session s=0x564ada8d10d0 osd=6 4 2024-05-15T12:45:14.238 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:12.864+0000 7fa39684b780 15 client.4429.objecter _session_op_assign 6 6 2024-05-15T12:45:14.238 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:12.864+0000 7fa39684b780 15 client.4429.objecter _send_op 6 to 6.1 on osd.6 2024-05-15T12:45:14.238 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:12.864+0000 7fa39684b780 1 -- 172.21.15.117:0/4241893909 --> [v2:172.21.15.178:6800/1383036222,v1:172.21.15.178:6801/1383036222] -- osd_op(unknown.0.0:6 6.1 6:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x564ada8b1190 con 0x564ada976cf0 2024-05-15T12:45:14.238 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:12.864+0000 7fa39684b780 20 client.4429.objecter put_session s=0x564ada8d10d0 osd=6 5 2024-05-15T12:45:14.238 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:12.864+0000 7fa39684b780 5 client.4429.objecter 2 in flight 2024-05-15T12:45:14.238 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:13.732+0000 7fa37bfff640 10 monclient: tick 2024-05-15T12:45:14.239 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:13.732+0000 7fa37bfff640 10 monclient: _check_auth_tickets 2024-05-15T12:45:14.239 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:13.732+0000 7fa37bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:45:14.239 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.246+0000 7fa3912ac640 1 -- 172.21.15.117:0/4241893909 <== osd.6 v2:172.21.15.178:6800/1383036222 5 ==== osd_map(41..42 src has 1..42) v4 ==== 1296+0+0 (crc 0 0 0) 0x7fa38c002050 con 0x564ada976cf0 2024-05-15T12:45:14.239 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.246+0000 7fa3912ac640 10 client.4429.objecter ms_dispatch 0x564ada7af9d0 osd_map(41..42 src has 1..42) v4 2024-05-15T12:45:14.239 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.246+0000 7fa3912ac640 3 client.4429.objecter handle_osd_map got epochs [41,42] > 40 2024-05-15T12:45:14.241 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.246+0000 7fa3912ac640 3 client.4429.objecter handle_osd_map decoding incremental epoch 41 2024-05-15T12:45:14.241 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.246+0000 7fa3912ac640 10 client.4429.objecter checking op 5 2024-05-15T12:45:14.241 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.246+0000 7fa3912ac640 20 client.4429.objecter _calc_target epoch 41 base exist @6 precalc_pgid 0 pgid 0.0 is_read 2024-05-15T12:45:14.241 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.246+0000 7fa3912ac640 20 client.4429.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-05-15T12:45:14.241 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.246+0000 7fa3912ac640 10 client.4429.objecter checking op 6 2024-05-15T12:45:14.242 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.246+0000 7fa3912ac640 20 client.4429.objecter _calc_target epoch 41 base noexist @6 precalc_pgid 0 pgid 0.0 is_read 2024-05-15T12:45:14.242 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.246+0000 7fa3912ac640 20 client.4429.objecter _calc_target target noexist @6 -> pgid 6.a7bbaa1 2024-05-15T12:45:14.242 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.246+0000 7fa3912ac640 3 client.4429.objecter handle_osd_map decoding incremental epoch 42 2024-05-15T12:45:14.242 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.246+0000 7fa3912ac640 10 client.4429.objecter checking op 5 2024-05-15T12:45:14.242 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.246+0000 7fa3912ac640 20 client.4429.objecter _calc_target epoch 42 base exist @6 precalc_pgid 0 pgid 0.0 is_read 2024-05-15T12:45:14.242 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.246+0000 7fa3912ac640 20 client.4429.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-05-15T12:45:14.242 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.246+0000 7fa3912ac640 10 client.4429.objecter checking op 6 2024-05-15T12:45:14.242 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.246+0000 7fa3912ac640 20 client.4429.objecter _calc_target epoch 42 base noexist @6 precalc_pgid 0 pgid 0.0 is_read 2024-05-15T12:45:14.242 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.246+0000 7fa3912ac640 20 client.4429.objecter _calc_target target noexist @6 -> pgid 6.a7bbaa1 2024-05-15T12:45:14.243 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.246+0000 7fa3912ac640 20 client.4429.objecter dump_active .. 0 homeless 2024-05-15T12:45:14.243 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.246+0000 7fa3912ac640 20 client.4429.objecter 5 6.d3a59201 osd.6 exist [read 0~256] 2024-05-15T12:45:14.243 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.246+0000 7fa3912ac640 20 client.4429.objecter 6 6.a7bbaa1 osd.6 noexist [read 0~256] 2024-05-15T12:45:14.243 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.251+0000 7fa3942b2640 1 -- 172.21.15.117:0/4241893909 <== osd.6 v2:172.21.15.178:6800/1383036222 6 ==== osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7fa38c001820 con 0x564ada976cf0 2024-05-15T12:45:14.243 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.251+0000 7fa3942b2640 10 client.4429.objecter ms_dispatch 0x564ada7af9d0 osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2024-05-15T12:45:14.243 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.251+0000 7fa3942b2640 10 client.4429.objecter in handle_osd_op_reply 2024-05-15T12:45:14.243 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.251+0000 7fa3942b2640 7 client.4429.objecter handle_osd_op_reply 5 ondisk uv 1 in 6.1 attempt 0 2024-05-15T12:45:14.243 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.251+0000 7fa3942b2640 10 client.4429.objecter op 0 rval 0 len 5 2024-05-15T12:45:14.243 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.251+0000 7fa3942b2640 15 client.4429.objecter handle_osd_op_reply completed tid 5 2024-05-15T12:45:14.243 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.251+0000 7fa3942b2640 15 client.4429.objecter _finish_op 5 2024-05-15T12:45:14.243 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.251+0000 7fa3942b2640 20 client.4429.objecter put_session s=0x564ada8d10d0 osd=6 5 2024-05-15T12:45:14.243 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.251+0000 7fa3942b2640 15 client.4429.objecter _session_op_remove 6 5 2024-05-15T12:45:14.243 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.251+0000 7fa3942b2640 5 client.4429.objecter 1 in flight 2024-05-15T12:45:14.243 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.251+0000 7fa3942b2640 1 -- 172.21.15.117:0/4241893909 <== osd.6 v2:172.21.15.178:6800/1383036222 7 ==== osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7fa38c001820 con 0x564ada976cf0 2024-05-15T12:45:14.243 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.251+0000 7fa3942b2640 10 client.4429.objecter ms_dispatch 0x564ada7af9d0 osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-05-15T12:45:14.244 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.251+0000 7fa3942b2640 10 client.4429.objecter in handle_osd_op_reply 2024-05-15T12:45:14.244 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.251+0000 7fa3942b2640 7 client.4429.objecter handle_osd_op_reply 6 ondisk uv 0 in 6.1 attempt 0 2024-05-15T12:45:14.244 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.251+0000 7fa3942b2640 10 client.4429.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-05-15T12:45:14.244 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.251+0000 7fa3942b2640 10 client.4429.objecter op 0 rval 0 len 0 2024-05-15T12:45:14.244 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.251+0000 7fa3942b2640 15 client.4429.objecter handle_osd_op_reply completed tid 6 2024-05-15T12:45:14.244 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.251+0000 7fa3942b2640 15 client.4429.objecter _finish_op 6 2024-05-15T12:45:14.244 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.251+0000 7fa3942b2640 20 client.4429.objecter put_session s=0x564ada8d10d0 osd=6 4 2024-05-15T12:45:14.244 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.251+0000 7fa3942b2640 15 client.4429.objecter _session_op_remove 6 6 2024-05-15T12:45:14.244 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.251+0000 7fa3942b2640 5 client.4429.objecter 0 in flight 2024-05-15T12:45:14.244 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (1387 ms) 2024-05-15T12:45:14.244 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2024-05-15T12:45:14.244 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.251+0000 7fa39684b780 10 client.4429.objecter _op_submit op 0x564ada976330 2024-05-15T12:45:14.244 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.251+0000 7fa39684b780 20 client.4429.objecter _calc_target epoch 42 base exist @6 precalc_pgid 0 pgid 0.0 is_read 2024-05-15T12:45:14.244 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.251+0000 7fa39684b780 20 client.4429.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-05-15T12:45:14.244 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.251+0000 7fa39684b780 10 client.4429.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [6,3] primary 6 2024-05-15T12:45:14.245 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.251+0000 7fa39684b780 20 client.4429.objecter _get_session s=0x564ada8d10d0 osd=6 3 2024-05-15T12:45:14.245 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.251+0000 7fa39684b780 10 client.4429.objecter _op_submit oid exist '@6' '@6' [read 0~256] tid 7 osd.6 2024-05-15T12:45:14.245 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.251+0000 7fa39684b780 20 client.4429.objecter get_session s=0x564ada8d10d0 osd=6 3 2024-05-15T12:45:14.245 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.251+0000 7fa39684b780 15 client.4429.objecter _session_op_assign 6 7 2024-05-15T12:45:14.245 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.251+0000 7fa39684b780 15 client.4429.objecter _send_op 7 to 6.1 on osd.6 2024-05-15T12:45:14.245 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.251+0000 7fa39684b780 1 -- 172.21.15.117:0/4241893909 --> [v2:172.21.15.178:6800/1383036222,v1:172.21.15.178:6801/1383036222] -- osd_op(unknown.0.0:7 6.1 6:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42) v9 -- 0x564ada8df960 con 0x564ada976cf0 2024-05-15T12:45:14.245 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.251+0000 7fa39684b780 20 client.4429.objecter put_session s=0x564ada8d10d0 osd=6 4 2024-05-15T12:45:14.245 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.251+0000 7fa39684b780 5 client.4429.objecter 1 in flight 2024-05-15T12:45:14.245 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.251+0000 7fa39684b780 10 client.4429.objecter _op_submit op 0x564ada8b1bf0 2024-05-15T12:45:14.245 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.251+0000 7fa39684b780 20 client.4429.objecter _calc_target epoch 42 base noexist @6 precalc_pgid 0 pgid 0.0 is_read 2024-05-15T12:45:14.245 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.251+0000 7fa39684b780 20 client.4429.objecter _calc_target target noexist @6 -> pgid 6.a7bbaa1 2024-05-15T12:45:14.245 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.251+0000 7fa39684b780 10 client.4429.objecter _calc_target raw pgid 6.a7bbaa1 -> actual 6.1 acting [6,3] primary 6 2024-05-15T12:45:14.245 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.251+0000 7fa39684b780 20 client.4429.objecter _get_session s=0x564ada8d10d0 osd=6 4 2024-05-15T12:45:14.245 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.251+0000 7fa39684b780 10 client.4429.objecter _op_submit oid noexist '@6' '@6' [read 0~256] tid 8 osd.6 2024-05-15T12:45:14.291 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.251+0000 7fa39684b780 20 client.4429.objecter get_session s=0x564ada8d10d0 osd=6 4 2024-05-15T12:45:14.292 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.251+0000 7fa39684b780 15 client.4429.objecter _session_op_assign 6 8 2024-05-15T12:45:14.292 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.251+0000 7fa39684b780 15 client.4429.objecter _send_op 8 to 6.1 on osd.6 2024-05-15T12:45:14.292 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.251+0000 7fa39684b780 1 -- 172.21.15.117:0/4241893909 --> [v2:172.21.15.178:6800/1383036222,v1:172.21.15.178:6801/1383036222] -- osd_op(unknown.0.0:8 6.1 6:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42) v9 -- 0x564ada8b20a0 con 0x564ada976cf0 2024-05-15T12:45:14.292 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.251+0000 7fa39684b780 20 client.4429.objecter put_session s=0x564ada8d10d0 osd=6 5 2024-05-15T12:45:14.292 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.251+0000 7fa39684b780 5 client.4429.objecter 2 in flight 2024-05-15T12:45:14.292 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.301+0000 7fa3942b2640 1 -- 172.21.15.117:0/4241893909 <== osd.6 v2:172.21.15.178:6800/1383036222 8 ==== osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7fa38c001820 con 0x564ada976cf0 2024-05-15T12:45:14.292 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.301+0000 7fa3942b2640 10 client.4429.objecter ms_dispatch 0x564ada7af9d0 osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2024-05-15T12:45:14.292 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.301+0000 7fa3942b2640 10 client.4429.objecter in handle_osd_op_reply 2024-05-15T12:45:14.292 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.301+0000 7fa3942b2640 7 client.4429.objecter handle_osd_op_reply 7 ondisk uv 1 in 6.1 attempt 0 2024-05-15T12:45:14.292 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.301+0000 7fa3942b2640 10 client.4429.objecter op 0 rval 0 len 5 2024-05-15T12:45:14.292 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.301+0000 7fa3942b2640 15 client.4429.objecter handle_osd_op_reply completed tid 7 2024-05-15T12:45:14.292 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.301+0000 7fa3942b2640 15 client.4429.objecter _finish_op 7 2024-05-15T12:45:14.293 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.301+0000 7fa3942b2640 20 client.4429.objecter put_session s=0x564ada8d10d0 osd=6 5 2024-05-15T12:45:14.293 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.301+0000 7fa3942b2640 15 client.4429.objecter _session_op_remove 6 7 2024-05-15T12:45:14.293 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.301+0000 7fa3942b2640 5 client.4429.objecter 1 in flight 2024-05-15T12:45:14.293 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.301+0000 7fa3942b2640 1 -- 172.21.15.117:0/4241893909 <== osd.6 v2:172.21.15.178:6800/1383036222 9 ==== osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7fa38c001820 con 0x564ada976cf0 2024-05-15T12:45:14.293 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.301+0000 7fa3942b2640 10 client.4429.objecter ms_dispatch 0x564ada7af9d0 osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-05-15T12:45:14.293 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.301+0000 7fa3942b2640 10 client.4429.objecter in handle_osd_op_reply 2024-05-15T12:45:14.293 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.301+0000 7fa3942b2640 7 client.4429.objecter handle_osd_op_reply 8 ondisk uv 0 in 6.1 attempt 0 2024-05-15T12:45:14.293 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.301+0000 7fa3942b2640 10 client.4429.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-05-15T12:45:14.293 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.301+0000 7fa3942b2640 10 client.4429.objecter op 0 rval 0 len 0 2024-05-15T12:45:14.293 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.301+0000 7fa3942b2640 15 client.4429.objecter handle_osd_op_reply completed tid 8 2024-05-15T12:45:14.293 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.301+0000 7fa3942b2640 15 client.4429.objecter _finish_op 8 2024-05-15T12:45:14.294 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.301+0000 7fa3942b2640 20 client.4429.objecter put_session s=0x564ada8d10d0 osd=6 4 2024-05-15T12:45:14.294 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.301+0000 7fa3942b2640 15 client.4429.objecter _session_op_remove 6 8 2024-05-15T12:45:14.294 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.301+0000 7fa3942b2640 5 client.4429.objecter 0 in flight 2024-05-15T12:45:14.294 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (50 ms) 2024-05-15T12:45:14.294 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2024-05-15T12:45:15.073 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:03:22.251 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:15.083 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:03:22.261 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:15.109 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:03:22.288 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:15.145 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:03:22.323 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:15.333 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:03:22.504 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:15.338 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:03:22.509 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:15.421 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:03:22.599 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:15.668 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:03:22.847 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:15.692 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:03:22.863 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:17.750 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.301+0000 7fa396avail 2024-05-15T12:45:17.750 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: pgs: 38.751% pgs unknown 2024-05-15T12:45:17.750 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 59.596% pgs not active 2024-05-15T12:45:17.750 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 553 creating+peering 2024-05-15T12:45:17.753 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 422 unknown 2024-05-15T12:45:17.753 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 96 creating+activating 2024-05-15T12:45:17.753 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 18 active+clean 2024-05-15T12:45:17.753 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 2024-05-15T12:45:17.753 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: io: 2024-05-15T12:45:17.753 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: client: 189 B/s wr, 0 op/s rd, 0 op/s wr 2024-05-15T12:45:17.753 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 2024-05-15T12:45:17.753 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 2024-05-15T12:45:17.753 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: [ OK ] LibRadosService.StatusFormat (5993 ms) 2024-05-15T12:45:17.753 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: [ RUN ] LibRadosService.Status 2024-05-15T12:45:17.754 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: [ OK ] LibRadosService.Status (20506 ms) 2024-05-15T12:45:17.754 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: [----------] 4 tests from LibRadosService (31887 ms total) 2024-05-15T12:45:17.754 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 2024-05-15T12:45:17.754 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: [----------] Global test environment tear-down 2024-05-15T12:45:17.754 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: [==========] 4 tests from 1 test suite ran. (31888 ms total) 2024-05-15T12:45:17.754 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: [ PASSED ] 4 tests. 2024-05-15T12:45:18.003 INFO:tasks.workunit.client.0.smithi117.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2024-05-15T12:45:18.003 INFO:tasks.workunit.client.0.smithi117.stdout: api_service_pp: [----------] Global test environment set-up. 2024-05-15T12:45:18.003 INFO:tasks.workunit.client.0.smithi117.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2024-05-15T12:45:18.004 INFO:tasks.workunit.client.0.smithi117.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2024-05-15T12:45:18.004 INFO:tasks.workunit.client.0.smithi117.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5491 ms) 2024-05-15T12:45:18.004 INFO:tasks.workunit.client.0.smithi117.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2024-05-15T12:45:18.004 INFO:tasks.workunit.client.0.smithi117.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (1041 ms) 2024-05-15T12:45:18.004 INFO:tasks.workunit.client.0.smithi117.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2024-05-15T12:45:18.004 INFO:tasks.workunit.client.0.smithi117.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20334 ms) 2024-05-15T12:45:18.004 INFO:tasks.workunit.client.0.smithi117.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2024-05-15T12:45:18.004 INFO:tasks.workunit.client.0.smithi117.stdout: api_service_pp: attempt 0 of 20 2024-05-15T12:45:18.004 INFO:tasks.workunit.client.0.smithi117.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (5300 ms) 2024-05-15T12:45:18.004 INFO:tasks.workunit.client.0.smithi117.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (32166 ms total) 2024-05-15T12:45:18.004 INFO:tasks.workunit.client.0.smithi117.stdout: api_service_pp: 2024-05-15T12:45:18.004 INFO:tasks.workunit.client.0.smithi117.stdout: api_service_pp: [----------] Global test environment tear-down 2024-05-15T12:45:18.004 INFO:tasks.workunit.client.0.smithi117.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (32166 ms total) 2024-05-15T12:45:18.004 INFO:tasks.workunit.client.0.smithi117.stdout: api_service_pp: [ PASSED ] 4 tests. 2024-05-15T12:45:19.033 INFO:tasks.ceph.osd.4.smithi178.stderr:2024-05-15T12:45:19.030+0000 12a65640 -1 osd.4 45 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4485.0:2 17.1a 17.db6ee63a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-05-15T12:45:19.265 INFO:tasks.ceph.osd.0.smithi117.stderr:2024-05-15T12:45:19.258+0000 12a65640 -1 osd.0 45 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4522.0:2 20.1 20.aabc5e21 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-05-15T12:45:19.723 INFO:tasks.workunit.client.0.smithi117.stdout:84b780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-05-15T12:45:19.723 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.301+0000 7fa39684b780 20 librados: queue_aio_write 0x564ada778830 completion 0x564ada9770d0 write_seq 1 2024-05-15T12:45:19.723 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.301+0000 7fa39684b780 10 client.4429.objecter _op_submit op 0x564ada976330 2024-05-15T12:45:19.723 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.301+0000 7fa39684b780 20 client.4429.objecter _calc_target epoch 42 base exist @6 precalc_pgid 0 pgid 0.0 is_write 2024-05-15T12:45:19.724 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.301+0000 7fa39684b780 20 client.4429.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-05-15T12:45:19.724 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.301+0000 7fa39684b780 10 client.4429.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [6,3] primary 6 2024-05-15T12:45:19.724 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.301+0000 7fa39684b780 20 client.4429.objecter _get_session s=0x564ada8d10d0 osd=6 3 2024-05-15T12:45:19.724 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.301+0000 7fa39684b780 10 client.4429.objecter _op_submit oid exist '@6' '@6' [write 0~5 in=5b] tid 9 osd.6 2024-05-15T12:45:19.724 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.301+0000 7fa39684b780 20 client.4429.objecter get_session s=0x564ada8d10d0 osd=6 3 2024-05-15T12:45:19.724 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.301+0000 7fa39684b780 15 client.4429.objecter _session_op_assign 6 9 2024-05-15T12:45:19.724 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.301+0000 7fa39684b780 15 client.4429.objecter _send_op 9 to 6.1 on osd.6 2024-05-15T12:45:19.724 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.301+0000 7fa39684b780 1 -- 172.21.15.117:0/4241893909 --> [v2:172.21.15.178:6800/1383036222,v1:172.21.15.178:6801/1383036222] -- osd_op(unknown.0.0:9 6.1 6:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42) v9 -- 0x564ada8b1580 con 0x564ada976cf0 2024-05-15T12:45:19.724 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.301+0000 7fa39684b780 20 client.4429.objecter put_session s=0x564ada8d10d0 osd=6 4 2024-05-15T12:45:19.724 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.301+0000 7fa39684b780 5 client.4429.objecter 1 in flight 2024-05-15T12:45:19.724 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.301+0000 7fa39684b780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-05-15T12:45:19.724 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.732+0000 7fa37bfff640 10 monclient: tick 2024-05-15T12:45:19.724 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.732+0000 7fa37bfff640 10 monclient: _check_auth_tickets 2024-05-15T12:45:19.724 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:14.732+0000 7fa37bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:45:19.724 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:15.732+0000 7fa37bfff640 10 monclient: tick 2024-05-15T12:45:19.725 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:15.732+0000 7fa37bfff640 10 monclient: _check_auth_tickets 2024-05-15T12:45:19.725 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:15.732+0000 7fa37bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:45:19.725 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:15.770+0000 7fa395d15640 10 client.4429.objecter tick 2024-05-15T12:45:19.725 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:16.733+0000 7fa37bfff640 10 monclient: tick 2024-05-15T12:45:19.725 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:16.733+0000 7fa37bfff640 10 monclient: _check_auth_tickets 2024-05-15T12:45:19.725 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:16.733+0000 7fa37bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:45:19.725 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:17.733+0000 7fa37bfff640 10 monclient: tick 2024-05-15T12:45:19.725 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:17.733+0000 7fa37bfff640 10 monclient: _check_auth_tickets 2024-05-15T12:45:19.725 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:17.733+0000 7fa37bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:45:19.725 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:18.733+0000 7fa37bfff640 10 monclient: tick 2024-05-15T12:45:19.725 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:18.733+0000 7fa37bfff640 10 monclient: _check_auth_tickets 2024-05-15T12:45:19.725 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:18.733+0000 7fa37bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:45:19.725 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:18.733+0000 7fa37bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:45:19.725 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:19.733+0000 7fa37bfff640 10 monclient: tick 2024-05-15T12:45:20.272 INFO:tasks.ceph.osd.4.smithi178.stderr:2024-05-15T12:45:20.199+0000 12a65640 -1 osd.4 45 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4485.0:2 17.1a 17.db6ee63a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-05-15T12:45:20.466 INFO:tasks.ceph.osd.0.smithi117.stderr:2024-05-15T12:45:20.425+0000 12a65640 -1 osd.0 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4522.0:2 20.1 20.aabc5e21 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-05-15T12:45:21.456 INFO:tasks.ceph.osd.0.smithi117.stderr:2024-05-15T12:45:21.451+0000 12a65640 -1 osd.0 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4522.0:2 20.1 20.aabc5e21 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-05-15T12:45:21.632 INFO:tasks.ceph.osd.4.smithi178.stderr:2024-05-15T12:45:21.505+0000 12a65640 -1 osd.4 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4485.0:2 17.1a 17.db6ee63a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-05-15T12:45:21.755 INFO:tasks.ceph.osd.7.smithi178.stderr:2024-05-15T12:45:21.748+0000 ea5d640 -1 osd.7 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4355.0:2 14.1f 14:fd6edea3:test-rados-api-smithi117-40175-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) 2024-05-15T12:45:22.513 INFO:tasks.ceph.osd.4.smithi178.stderr:2024-05-15T12:45:22.507+0000 12a65640 -1 osd.4 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4485.0:2 17.1a 17.db6ee63a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-05-15T12:45:22.667 INFO:tasks.ceph.osd.0.smithi117.stderr:2024-05-15T12:45:22.613+0000 12a65640 -1 osd.0 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4522.0:2 20.1 20.aabc5e21 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-05-15T12:45:22.702 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-05-15T12:45:22.673+0000 12a65640 -1 osd.2 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4561.0:2 34.3 34:c64b540a:test-rados-api-smithi117-40597-3::testobj:head [write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-05-15T12:45:22.798 INFO:tasks.ceph.osd.7.smithi178.stderr:2024-05-15T12:45:22.800+0000 ea5d640 -1 osd.7 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4355.0:2 14.1f 14:fd6edea3:test-rados-api-smithi117-40175-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) 2024-05-15T12:45:23.108 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:19.733+0000 7fa37bfff640 10 monclient: _check_auth_tickets 2024-05-15T12:45:23.108 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:19.733+0000 7fa37bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:45:23.108 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:20.733+0000 7fa37bfff640 10 monclient: tick 2024-05-15T12:45:23.109 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:20.733+0000 7fa37bfff640 10 monclient: _check_auth_tickets 2024-05-15T12:45:23.109 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:20.733+0000 7fa37bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:45:23.109 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:20.771+0000 7fa395d15640 10 client.4429.objecter tick 2024-05-15T12:45:23.109 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:21.733+0000 7fa37bfff640 10 monclient: tick 2024-05-15T12:45:23.109 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:21.733+0000 7fa37bfff640 10 monclient: _check_auth_tickets 2024-05-15T12:45:23.109 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:21.733+0000 7fa37bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:45:23.109 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:22.733+0000 7fa37bfff640 10 monclient: tick 2024-05-15T12:45:23.109 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:22.733+0000 7fa37bfff640 10 monclient: _check_auth_tickets 2024-05-15T12:45:23.109 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:22.733+0000 7fa37bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:45:23.109 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:23.118+0000 7fa3942b2640 1 -- 172.21.15.117:0/4241893909 <== osd.6 v2:172.21.15.178:6800/1383036222 10 ==== osd_op_reply(9 exist [write 0~5] v42'2 uv2 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7fa38c001820 con 0x564ada976cf0 2024-05-15T12:45:23.109 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:23.118+0000 7fa3942b2640 10 client.4429.objecter ms_dispatch 0x564ada7af9d0 osd_op_reply(9 exist [write 0~5] v42'2 uv2 ondisk = 0) v8 2024-05-15T12:45:23.109 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:23.118+0000 7fa3942b2640 10 client.4429.objecter in handle_osd_op_reply 2024-05-15T12:45:23.109 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:23.118+0000 7fa3942b2640 7 client.4429.objecter handle_osd_op_reply 9 ondisk uv 2 in 6.1 attempt 0 2024-05-15T12:45:23.109 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:23.118+0000 7fa3942b2640 10 client.4429.objecter op 0 rval 0 len 0 2024-05-15T12:45:23.109 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:23.118+0000 7fa3942b2640 15 client.4429.objecter handle_osd_op_reply completed tid 9 2024-05-15T12:45:23.110 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:23.118+0000 7fa3942b2640 15 client.4429.objecter _finish_op 9 2024-05-15T12:45:23.110 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:23.118+0000 7fa3942b2640 20 client.4429.objecter put_session s=0x564ada8d10d0 osd=6 4 2024-05-15T12:45:23.110 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:23.118+0000 7fa3942b2640 15 client.4429.objecter _session_op_remove 6 9 2024-05-15T12:45:23.110 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:23.118+0000 7fa3942b2640 5 client.4429.objecter 0 in flight 2024-05-15T12:45:23.110 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:23.118+0000 7fa3942b2640 20 librados: complete_aio_write 0x564ada9770d0 2024-05-15T12:45:23.110 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (8817 ms) 2024-05-15T12:45:23.110 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2024-05-15T12:45:23.110 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:23.118+0000 7fa39684b780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-05-15T12:45:23.110 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:23.118+0000 7fa39684b780 20 librados: queue_aio_write 0x564ada778830 completion 0x564ada75d480 write_seq 2 2024-05-15T12:45:23.110 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:23.118+0000 7fa39684b780 10 client.4429.objecter _op_submit op 0x564ada976330 2024-05-15T12:45:23.110 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:23.118+0000 7fa39684b780 20 client.4429.objecter _calc_target epoch 42 base exist @6 precalc_pgid 0 pgid 0.0 is_write 2024-05-15T12:45:23.110 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:23.118+0000 7fa39684b780 20 client.4429.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-05-15T12:45:23.110 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:23.118+0000 7fa39684b780 10 client.4429.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [6,3] primary 6 2024-05-15T12:45:23.110 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:23.118+0000 7fa39684b780 20 client.4429.objecter _get_session s=0x564ada8d10d0 osd=6 3 2024-05-15T12:45:23.567 INFO:tasks.ceph.osd.4.smithi178.stderr:2024-05-15T12:45:23.532+0000 12a65640 -1 osd.4 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4485.0:2 17.1a 17:5c6776db:::0.obj:head [write 0~9 in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-05-15T12:45:23.766 INFO:tasks.ceph.osd.0.smithi117.stderr:2024-05-15T12:45:23.747+0000 12a65640 -1 osd.0 47 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4522.0:2 20.1 20.aabc5e21 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-05-15T12:45:23.919 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-05-15T12:45:23.780+0000 12a65640 -1 osd.2 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4561.0:2 34.3 34:c64b540a:test-rados-api-smithi117-40597-3::testobj:head [write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-05-15T12:45:23.963 INFO:tasks.ceph.osd.7.smithi178.stderr:2024-05-15T12:45:23.969+0000 ea5d640 -1 osd.7 47 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4355.0:2 14.1f 14:fd6edea3:test-rados-api-smithi117-40175-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) 2024-05-15T12:45:24.543 INFO:tasks.ceph.osd.4.smithi178.stderr:2024-05-15T12:45:24.544+0000 12a65640 -1 osd.4 46 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4485.0:2 17.1a 17:5c6776db:::0.obj:head [write 0~9 in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-05-15T12:45:24.879 INFO:tasks.ceph.osd.0.smithi117.stderr:2024-05-15T12:45:24.859+0000 12a65640 -1 osd.0 47 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4522.0:2 20.1 20.aabc5e21 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-05-15T12:45:25.422 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:23.118+0000 7fa39684b780 10 client.4429.objecter _op_submit oid exist '@6' '@6' [write 0~5 in=5 api_misc: [==========] Running 12 tests from 4 test suites. 2024-05-15T12:45:25.422 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [----------] Global test environment set-up. 2024-05-15T12:45:25.422 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2024-05-15T12:45:25.422 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2024-05-15T12:45:25.422 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2024-05-15T12:45:25.422 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-05-15T12:45:25.422 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: 2024-05-15T12:45:25.422 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2024-05-15T12:45:25.422 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2024-05-15T12:45:25.422 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2024-05-15T12:45:25.422 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: 2024-05-15T12:44:46.528+0000 7f78f463c780 -1 failed for service _ceph-mon._tcp 2024-05-15T12:45:25.422 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: 2024-05-15T12:44:46.528+0000 7f78f463c780 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-05-15T12:45:25.422 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (859 ms) 2024-05-15T12:45:25.423 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2024-05-15T12:45:25.423 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (2005 ms) 2024-05-15T12:45:25.423 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (2864 ms total) 2024-05-15T12:45:25.423 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: 2024-05-15T12:45:25.423 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2024-05-15T12:45:25.423 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2024-05-15T12:45:25.423 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: started 0x5646fc9127b0 2024-05-15T12:45:25.423 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: started 0x5646fcb1f830 2024-05-15T12:45:25.423 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: started 0x5646fd549390 2024-05-15T12:45:25.423 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: started 0x5646fd5463d0 2024-05-15T12:45:25.423 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: started 0x5646fcc968b0 2024-05-15T12:45:25.423 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: started 0x5646fcc97310 2024-05-15T12:45:25.423 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: started 0x5646fcc97d90 2024-05-15T12:45:25.423 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: started 0x5646fcc98830 2024-05-15T12:45:25.423 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: started 0x5646fcc992a0 2024-05-15T12:45:25.424 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: started 0x5646fcbefa20 2024-05-15T12:45:25.424 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: started 0x5646fcc99e20 2024-05-15T12:45:25.424 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: started 0x5646fcc9a8e0 2024-05-15T12:45:25.424 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: started 0x5646fcc99060 2024-05-15T12:45:25.424 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: started 0x5646fd540250 2024-05-15T12:45:25.424 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: started 0x5646fcc9a660 2024-05-15T12:45:25.424 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: started 0x5646fd541100 2024-05-15T12:45:25.424 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: started 0x5646fd53ff20 2024-05-15T12:45:25.424 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: started 0x5646fd541c60 2024-05-15T12:45:25.424 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: started 0x5646fd542740 2024-05-15T12:45:25.424 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: started 0x5646fd540e70 2024-05-15T12:45:25.424 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: started 0x5646fd5432d0 2024-05-15T12:45:25.424 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: started 0x5646fd543dc0 2024-05-15T12:45:25.424 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: started 0x5646fd544060 2024-05-15T12:45:25.424 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: started 0x5646fd542450 2024-05-15T12:45:25.424 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: started 0x5646fcc9c600 2024-05-15T12:45:25.425 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: started 0x5646fd543ab0 2024-05-15T12:45:25.425 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: started 0x5646fcc9d140 2024-05-15T12:45:25.425 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: started 0x5646fcc9dbd0 2024-05-15T12:45:25.425 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: started 0x5646fcc9c300 2024-05-15T12:45:25.425 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: started 0x5646fcc9e730 2024-05-15T12:45:25.425 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: started 0x5646fcc9f1c0 2024-05-15T12:45:25.425 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: started 0x5646fcc9d930 2024-05-15T12:45:25.425 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: started 32 aios 2024-05-15T12:45:25.425 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: waiting 0x5646fc9127b0 2024-05-15T12:45:25.425 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: waiting 0x5646fcb1f830 2024-05-15T12:45:25.425 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: waiting 0x5646fd549390 2024-05-15T12:45:25.425 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: waiting 0x5646fd5463d0 2024-05-15T12:45:25.425 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: waiting 0x5646fcc968b0 2024-05-15T12:45:25.425 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: waiting 0x5646fcc97310 2024-05-15T12:45:25.425 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: waiting 0x5646fcc97d90 2024-05-15T12:45:25.426 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: waiting 0x5646fcc98830 2024-05-15T12:45:25.426 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: waiting 0x5646fcc992a0 2024-05-15T12:45:25.426 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: waiting 0x5646fcbefa20 2024-05-15T12:45:25.426 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: waiting 0x5646fcc99e20 2024-05-15T12:45:25.426 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: waiting 0x5646fcc9a8e0 2024-05-15T12:45:25.426 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: waiting 0x5646fcc99060 2024-05-15T12:45:25.426 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: waiting 0x5646fd540250 2024-05-15T12:45:25.426 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: waiting 0x5646fcc9a660 2024-05-15T12:45:25.426 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: waiting 0x5646fd541100 2024-05-15T12:45:25.426 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: waiting 0x5646fd53ff20 2024-05-15T12:45:25.426 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: waiting 0x5646fd541c60 2024-05-15T12:45:25.596 INFO:tasks.ceph.osd.4.smithi178.stderr:2024-05-15T12:45:25.599+0000 12a65640 -1 osd.4 47 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4485.0:2 17.1a 17:5c6776db:::0.obj:head [write 0~9 in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-05-15T12:45:25.856 INFO:tasks.ceph.osd.0.smithi117.stderr:2024-05-15T12:45:25.860+0000 12a65640 -1 osd.0 47 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4522.0:2 20.1 20:847a3d55:::obj:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-05-15T12:45:26.368 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: waiting 0x56b] tid 10 osd.6 2024-05-15T12:45:26.368 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:23.118+0000 7fa39684b780 20 client.4429.objecter get_session s=0x564ada8d10d0 osd=6 3 2024-05-15T12:45:26.368 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:23.118+0000 7fa39684b780 15 client.4429.objecter _session_op_assign 6 10 2024-05-15T12:45:26.368 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:23.118+0000 7fa39684b780 15 client.4429.objecter _send_op 10 to 6.1 on osd.6 2024-05-15T12:45:26.368 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:23.118+0000 7fa39684b780 1 -- 172.21.15.117:0/4241893909 --> [v2:172.21.15.178:6800/1383036222,v1:172.21.15.178:6801/1383036222] -- osd_op(unknown.0.0:10 6.1 6:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42) v9 -- 0x564ada8b1580 con 0x564ada976cf0 2024-05-15T12:45:26.368 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:23.118+0000 7fa39684b780 20 client.4429.objecter put_session s=0x564ada8d10d0 osd=6 4 2024-05-15T12:45:26.369 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:23.118+0000 7fa39684b780 5 client.4429.objecter 1 in flight 2024-05-15T12:45:26.369 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:23.118+0000 7fa39684b780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-05-15T12:45:26.369 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:23.733+0000 7fa37bfff640 10 monclient: tick 2024-05-15T12:45:26.369 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:23.733+0000 7fa37bfff640 10 monclient: _check_auth_tickets 2024-05-15T12:45:26.369 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:23.733+0000 7fa37bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:45:26.369 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:24.733+0000 7fa37bfff640 10 monclient: tick 2024-05-15T12:45:26.369 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:24.733+0000 7fa37bfff640 10 monclient: _check_auth_tickets 2024-05-15T12:45:26.369 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:24.733+0000 7fa37bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:45:26.369 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:25.733+0000 7fa37bfff640 10 monclient: tick 2024-05-15T12:45:26.369 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:25.733+0000 7fa37bfff640 10 monclient: _check_auth_tickets 2024-05-15T12:45:26.369 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:25.733+0000 7fa37bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:45:26.369 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:25.772+0000 7fa395d15640 10 client.4429.objecter tick 2024-05-15T12:45:26.369 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.377+0000 7fa3912ac640 1 -- 172.21.15.117:0/4241893909 <== osd.6 v2:172.21.15.178:6800/1383036222 11 ==== osd_map(43..47 src has 1..47) v4 ==== 3614+0+0 (crc 0 0 0) 0x7fa38c001ae0 con 0x564ada976cf0 2024-05-15T12:45:26.370 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.377+0000 7fa3912ac640 10 client.4429.objecter ms_dispatch 0x564ada7af9d0 osd_map(43..47 src has 1..47) v4 2024-05-15T12:45:26.370 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.377+0000 7fa3912ac640 3 client.4429.objecter handle_osd_map got epochs [43,47] > 42 2024-05-15T12:45:26.370 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.377+0000 7fa3912ac640 3 client.4429.objecter handle_osd_map decoding incremental epoch 43 2024-05-15T12:45:26.370 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.377+0000 7fa3912ac640 10 client.4429.objecter checking op 10 2024-05-15T12:45:26.370 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.377+0000 7fa3912ac640 20 client.4429.objecter _calc_target epoch 43 base exist @6 precalc_pgid 0 pgid 0.0 is_write 2024-05-15T12:45:26.370 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.377+0000 7fa3912ac640 20 client.4429.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-05-15T12:45:26.370 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.377+0000 7fa3912ac640 3 client.4429.objecter handle_osd_map decoding incremental epoch 44 2024-05-15T12:45:26.370 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.377+0000 7fa3912ac640 10 client.4429.objecter checking op 10 2024-05-15T12:45:26.370 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.377+0000 7fa3912ac640 20 client.4429.objecter _calc_target epoch 44 base exist @6 precalc_pgid 0 pgid 0.0 is_write 2024-05-15T12:45:26.370 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.377+0000 7fa3912ac640 20 client.4429.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-05-15T12:45:26.370 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.377+0000 7fa3912ac640 3 client.4429.objecter handle_osd_map decoding incremental epoch 45 2024-05-15T12:45:26.370 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.377+0000 7fa3912ac640 10 client.4429.objecter checking op 10 2024-05-15T12:45:26.370 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.377+0000 7fa3912ac640 20 client.4429.objecter _calc_target epoch 45 base exist @6 precalc_pgid 0 pgid 0.0 is_write 2024-05-15T12:45:26.370 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.377+0000 7fa3912ac640 20 client.4429.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-05-15T12:45:26.370 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.377+0000 7fa3942b2640 1 -- 172.21.15.117:0/4241893909 <== osd.6 v2:172.21.15.178:6800/1383036222 12 ==== osd_op_reply(10 exist [write 0~5] v47'3 uv3 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7fa38c001820 con 0x564ada976cf0 2024-05-15T12:45:26.371 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.377+0000 7fa3912ac640 3 client.4429.objecter handle_osd_map decoding incremental epoch 46 2024-05-15T12:45:26.371 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.377+0000 7fa3942b2640 10 client.4429.objecter ms_dispatch 0x564ada7af9d0 osd_op_reply(10 exist [write 0~5] v47'3 uv3 ondisk = 0) v8 2024-05-15T12:45:26.371 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.377+0000 7fa3942b2640 10 client.4429.objecter in handle_osd_op_reply 2024-05-15T12:45:26.371 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.377+0000 7fa3912ac640 10 client.4429.objecter checking op 10 2024-05-15T12:45:26.371 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.377+0000 7fa3912ac640 20 client.4429.objecter _calc_target epoch 46 base exist @6 precalc_pgid 0 pgid 0.0 is_write 2024-05-15T12:45:26.371 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.377+0000 7fa3912ac640 20 client.4429.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-05-15T12:45:26.371 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.377+0000 7fa3912ac640 3 client.4429.objecter handle_osd_map decoding incremental epoch 47 2024-05-15T12:45:26.371 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.377+0000 7fa3912ac640 10 client.4429.objecter checking op 10 2024-05-15T12:45:26.371 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.377+0000 7fa3912ac640 20 client.4429.objecter _calc_target epoch 47 base exist @6 precalc_pgid 0 pgid 0.0 is_write 2024-05-15T12:45:26.371 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.377+0000 7fa3912ac640 20 client.4429.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-05-15T12:45:26.371 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.377+0000 7fa3912ac640 20 client.4429.objecter dump_active .. 0 homeless 2024-05-15T12:45:26.371 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.377+0000 7fa3912ac640 20 client.4429.objecter 10 6.d3a59201 osd.6 exist [write 0~5 in=5b] 2024-05-15T12:45:26.371 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.377+0000 7fa3942b2640 7 client.4429.objecter handle_osd_op_reply 10 ondisk uv 3 in 6.1 attempt 0 2024-05-15T12:45:26.371 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.377+0000 7fa3942b2640 10 client.4429.objecter op 0 rval 0 len 0 2024-05-15T12:45:26.371 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.377+0000 7fa3942b2640 15 client.4429.objecter handle_osd_op_reply completed tid 10 2024-05-15T12:45:26.372 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.377+0000 7fa3942b2640 15 client.4429.objecter _finish_op 10 2024-05-15T12:45:26.372 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.377+0000 7fa3942b2640 20 client.4429.objecter put_session s=0x564ada8d10d0 osd=6 4 2024-05-15T12:45:26.372 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.377+0000 7fa3942b2640 15 client.4429.objecter _session_op_remove 6 10 2024-05-15T12:45:26.372 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.377+0000 7fa3942b2640 5 client.4429.objecter 0 in flight 2024-05-15T12:45:26.372 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.377+0000 7fa3942b2640 20 librados: complete_aio_write 0x564ada75d480 2024-05-15T12:45:26.372 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (3260 ms) 2024-05-15T12:45:26.372 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2024-05-15T12:45:26.372 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.378+0000 7fa39684b780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-05-15T12:45:26.372 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.378+0000 7fa39684b780 20 librados: queue_aio_write 0x564ada778830 completion 0x564ada8b1760 write_seq 3 2024-05-15T12:45:26.372 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.378+0000 7fa39684b780 10 client.4429.objecter _op_submit op 0x564ada976330 2024-05-15T12:45:26.372 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.378+0000 7fa39684b780 20 client.4429.objecter _calc_target epoch 47 base exist @6 precalc_pgid 0 pgid 0.0 is_write 2024-05-15T12:45:26.372 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.378+0000 7fa39684b780 20 client.4429.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-05-15T12:45:26.555 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.378+0000 7fa39684b780 10 client.4429.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [6,3] primary 6 2024-05-15T12:45:26.556 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.378+0000 7fa39684b780 20 client.4429.objecter _get_session s=0x564ada8d10d0 osd=6 3 2024-05-15T12:45:26.556 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.378+0000 7fa39684b780 10 client.4429.objecter _op_submit oid exist '@6' '@6' [write 0~5 in=5b] tid 11 osd.6 2024-05-15T12:45:26.556 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.378+0000 7fa39684b780 20 client.4429.objecter get_session s=0x564ada8d10d0 osd=6 3 2024-05-15T12:45:26.556 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.378+0000 7fa39684b780 15 client.4429.objecter _session_op_assign 6 11 2024-05-15T12:45:26.556 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.378+0000 7fa39684b780 15 client.4429.objecter _send_op 11 to 6.1 on osd.6 2024-05-15T12:45:26.556 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.378+0000 7fa39684b780 1 -- 172.21.15.117:0/4241893909 --> [v2:172.21.15.178:6800/1383036222,v1:172.21.15.178:6801/1383036222] -- osd_op(unknown.0.0:11 6.1 6:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47) v9 -- 0x564ada8b5620 con 0x564ada976cf0 2024-05-15T12:45:26.556 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.378+0000 7fa39684b780 20 client.4429.objecter put_session s=0x564ada8d10d0 osd=6 4 2024-05-15T12:45:26.556 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.378+0000 7fa39684b780 5 client.4429.objecter 1 in flight 2024-05-15T12:45:26.556 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.378+0000 7fa39684b780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-05-15T12:45:26.556 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.565+0000 7fa3942b2640 1 -- 172.21.15.117:0/4241893909 <== osd.6 v2:172.21.15.178:6800/1383036222 13 ==== osd_op_reply(11 exist [write 0~5] v47'4 uv4 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7fa38c001820 con 0x564ada976cf0 2024-05-15T12:45:26.556 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.565+0000 7fa3942b2640 10 client.4429.objecter ms_dispatch 0x564ada7af9d0 osd_op_reply(11 exist [write 0~5] v47'4 uv4 ondisk = 0) v8 2024-05-15T12:45:26.556 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.565+0000 7fa3942b2640 10 client.4429.objecter in handle_osd_op_reply 2024-05-15T12:45:26.556 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.565+0000 7fa3942b2640 7 client.4429.objecter handle_osd_op_reply 11 ondisk uv 4 in 6.1 attempt 0 2024-05-15T12:45:26.557 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.565+0000 7fa3942b2640 10 client.4429.objecter op 0 rval 0 len 0 2024-05-15T12:45:26.557 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.565+0000 7fa3942b2640 15 client.4429.objecter handle_osd_op_reply completed tid 11 2024-05-15T12:45:26.557 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.565+0000 7fa3942b2640 15 client.4429.objecter _finish_op 11 2024-05-15T12:45:26.557 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.565+0000 7fa3942b2640 20 client.4429.objecter put_session s=0x564ada8d10d0 osd=6 4 2024-05-15T12:45:26.557 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.565+0000 7fa3942b2640 15 client.4429.objecter _session_op_remove 6 11 2024-05-15T12:45:26.557 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.565+0000 7fa3942b2640 5 client.4429.objecter 0 in flight 2024-05-15T12:45:26.557 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.565+0000 7fa3942b2640 20 librados: complete_aio_write 0x564ada8b1760 2024-05-15T12:45:26.557 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (187 ms) 2024-05-15T12:45:26.557 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2024-05-15T12:45:26.557 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.565+0000 7fa39684b780 10 client.4429.objecter _op_submit op 0x564ada976330 2024-05-15T12:45:26.557 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.565+0000 7fa39684b780 20 client.4429.objecter _calc_target epoch 47 base exist @6 precalc_pgid 0 pgid 0.0 is_read 2024-05-15T12:45:26.557 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.565+0000 7fa39684b780 20 client.4429.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-05-15T12:45:26.557 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.565+0000 7fa39684b780 10 client.4429.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [6,3] primary 6 2024-05-15T12:45:26.557 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.565+0000 7fa39684b780 20 client.4429.objecter _get_session s=0x564ada8d10d0 osd=6 3 2024-05-15T12:45:26.557 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.565+0000 7fa39684b780 10 client.4429.objecter _op_submit oid exist '@6' '@6' [read 0~0] tid 12 osd.6 2024-05-15T12:45:26.558 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.565+0000 7fa39684b780 20 client.4429.objecter get_session s=0x564ada8d10d0 osd=6 3 2024-05-15T12:45:26.571 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.565+0000 7fa39684b780 15 client.4429.objecter _session_op_assign 6 12 2024-05-15T12:45:26.571 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.565+0000 7fa39684b780 15 client.4429.objecter _send_op 12 to 6.1 on osd.6 2024-05-15T12:45:26.571 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.565+0000 7fa39684b780 1 -- 172.21.15.117:0/4241893909 --> [v2:172.21.15.178:6800/1383036222,v1:172.21.15.178:6801/1383036222] -- osd_op(unknown.0.0:12 6.1 6:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47) v9 -- 0x564ada8b4fb0 con 0x564ada976cf0 2024-05-15T12:45:26.571 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.565+0000 7fa39684b780 20 client.4429.objecter put_session s=0x564ada8d10d0 osd=6 4 2024-05-15T12:45:26.571 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.565+0000 7fa39684b780 5 client.4429.objecter 1 in flight 2024-05-15T12:45:26.571 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.565+0000 7fa39684b780 10 client.4429.objecter _op_submit op 0x564ada8b5350 2024-05-15T12:45:26.572 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.565+0000 7fa39684b780 20 client.4429.objecter _calc_target epoch 47 base noexist @6 precalc_pgid 0 pgid 0.0 is_read 2024-05-15T12:45:26.572 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.565+0000 7fa39684b780 20 client.4429.objecter _calc_target target noexist @6 -> pgid 6.a7bbaa1 2024-05-15T12:45:26.572 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.565+0000 7fa39684b780 10 client.4429.objecter _calc_target raw pgid 6.a7bbaa1 -> actual 6.1 acting [6,3] primary 6 2024-05-15T12:45:26.572 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.565+0000 7fa39684b780 20 client.4429.objecter _get_session s=0x564ada8d10d0 osd=6 4 2024-05-15T12:45:26.572 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.565+0000 7fa39684b780 10 client.4429.objecter _op_submit oid noexist '@6' '@6' [read 0~0] tid 13 osd.6 2024-05-15T12:45:26.572 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.565+0000 7fa39684b780 20 client.4429.objecter get_session s=0x564ada8d10d0 osd=6 4 2024-05-15T12:45:26.572 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.565+0000 7fa39684b780 15 client.4429.objecter _session_op_assign 6 13 2024-05-15T12:45:26.572 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.565+0000 7fa39684b780 15 client.4429.objecter _send_op 13 to 6.1 on osd.6 2024-05-15T12:45:26.572 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.565+0000 7fa39684b780 1 -- 172.21.15.117:0/4241893909 --> [v2:172.21.15.178:6800/1383036222,v1:172.21.15.178:6801/1383036222] -- osd_op(unknown.0.0:13 6.1 6:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47) v9 -- 0x564ada8b2440 con 0x564ada976cf0 2024-05-15T12:45:26.572 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.565+0000 7fa39684b780 20 client.4429.objecter put_session s=0x564ada8d10d0 osd=6 5 2024-05-15T12:45:26.572 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.565+0000 7fa39684b780 5 client.4429.objecter 2 in flight 2024-05-15T12:45:26.572 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.575+0000 7fa3942b2640 1 -- 172.21.15.117:0/4241893909 <== osd.6 v2:172.21.15.178:6800/1383036222 14 ==== osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7fa38c001820 con 0x564ada976cf0 2024-05-15T12:45:26.572 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.575+0000 7fa3942b2640 10 client.4429.objecter ms_dispatch 0x564ada7af9d0 osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2024-05-15T12:45:26.572 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.575+0000 7fa3942b2640 10 client.4429.objecter in handle_osd_op_reply 2024-05-15T12:45:26.572 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.575+0000 7fa3942b2640 7 client.4429.objecter handle_osd_op_reply 12 ondisk uv 4 in 6.1 attempt 0 2024-05-15T12:45:26.573 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.575+0000 7fa3942b2640 10 client.4429.objecter op 0 rval 0 len 5 2024-05-15T12:45:26.573 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.575+0000 7fa3942b2640 15 client.4429.objecter handle_osd_op_reply completed tid 12 2024-05-15T12:45:26.573 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.575+0000 7fa3942b2640 15 client.4429.objecter _finish_op 12 2024-05-15T12:45:26.573 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.575+0000 7fa3942b2640 20 client.4429.objecter put_session s=0x564ada8d10d0 osd=6 5 2024-05-15T12:45:26.573 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.575+0000 7fa3942b2640 15 client.4429.objecter _session_op_remove 6 12 2024-05-15T12:45:26.573 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.575+0000 7fa3942b2640 5 client.4429.objecter 1 in flight 2024-05-15T12:45:26.573 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.581+0000 7fa3942b2640 1 -- 172.21.15.117:0/4241893909 <== osd.6 v2:172.21.15.178:6800/1383036222 15 ==== osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7fa38c001820 con 0x564ada976cf0 2024-05-15T12:45:26.573 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.581+0000 7fa3942b2640 10 client.4429.objecter ms_dispatch 0x564ada7af9d0 osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-05-15T12:45:26.573 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.581+0000 7fa3942b2640 10 client.4429.objecter in handle_osd_op_reply 2024-05-15T12:45:26.573 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.581+0000 7fa3942b2640 7 client.4429.objecter handle_osd_op_reply 13 ondisk uv 0 in 6.1 attempt 0 2024-05-15T12:45:26.573 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.581+0000 7fa3942b2640 10 client.4429.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-05-15T12:45:26.573 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.581+0000 7fa3942b2640 10 client.4429.objecter op 0 rval 0 len 0 2024-05-15T12:45:26.573 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.581+0000 7fa3942b2640 15 client.4429.objecter handle_osd_op_reply completed tid 13 2024-05-15T12:45:26.573 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.581+0000 7fa3942b2640 15 client.4429.objecter _finish_op 13 2024-05-15T12:45:26.573 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.581+0000 7fa3942b2640 20 client.4429.objecter put_session s=0x564ada8d10d0 osd=6 4 2024-05-15T12:45:26.573 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.581+0000 7fa3942b2640 15 client.4429.objecter _session_op_remove 6 13 2024-05-15T12:45:26.574 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.581+0000 7fa3942b2640 5 client.4429.objecter 0 in flight 2024-05-15T12:45:26.574 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (16 ms) 2024-05-15T12:45:26.574 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2024-05-15T12:45:26.574 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.581+0000 7fa39684b780 10 client.4429.objecter _op_submit op 0x564ada8b4fb0 2024-05-15T12:45:26.574 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.581+0000 7fa39684b780 20 client.4429.objecter _calc_target epoch 47 base exist @6 precalc_pgid 0 pgid 0.0 is_read 2024-05-15T12:45:26.574 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.581+0000 7fa39684b780 20 client.4429.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-05-15T12:45:26.574 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.581+0000 7fa39684b780 10 client.4429.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [6,3] primary 6 2024-05-15T12:45:26.574 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.581+0000 7fa39684b780 20 client.4429.objecter _get_session s=0x564ada8d10d0 osd=6 3 2024-05-15T12:45:26.574 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.581+0000 7fa39684b780 10 client.4429.objecter _op_submit oid exist '@6' '@6' [read 0~0] tid 14 osd.6 2024-05-15T12:45:26.574 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.581+0000 7fa39684b780 20 client.4429.objecter get_session s=0x564ada8d10d0 osd=6 3 2024-05-15T12:45:26.574 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.581+0000 7fa39684b780 15 client.4429.objecter _session_op_assign 6 14 2024-05-15T12:45:26.574 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.581+0000 7fa39684b780 15 client.4429.objecter _send_op 14 to 6.1 on osd.6 2024-05-15T12:45:26.574 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.581+0000 7fa39684b780 1 -- 172.21.15.117:0/4241893909 --> [v2:172.21.15.178:6800/1383036222,v1:172.21.15.178:6801/1383036222] -- osd_op(unknown.0.0:14 6.1 6:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47) v9 -- 0x564ada976440 con 0x564ada976cf0 2024-05-15T12:45:26.574 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.581+0000 7fa39684b780 20 client.4429.objecter put_session s=0x564ada8d10d0 osd=6 4 2024-05-15T12:45:26.574 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.581+0000 7fa39684b780 5 client.4429.objecter 1 in flight 2024-05-15T12:45:26.574 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.581+0000 7fa39684b780 10 client.4429.objecter _op_submit op 0x564ada8b2440 2024-05-15T12:45:26.575 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.581+0000 7fa39684b780 20 client.4429.objecter _calc_target epoch 47 base noexist @6 precalc_pgid 0 pgid 0.0 is_read 2024-05-15T12:45:26.575 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.581+0000 7fa39684b780 20 client.4429.objecter _calc_target target noexist @6 -> pgid 6.a7bbaa1 2024-05-15T12:45:26.595 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.581+0000 7fa39684b780 10 client.4429.objecter _calc_target raw pgid 6.a7bbaa1 -> actual 6.1 acting [6,3] primary 6 2024-05-15T12:45:26.595 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.581+0000 7fa39684b780 20 client.4429.objecter _get_session s=0x564ada8d10d0 osd=6 4 2024-05-15T12:45:26.595 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.581+0000 7fa39684b780 10 client.4429.objecter _op_submit oid noexist '@6' '@6' [read 0~0] tid 15 osd.6 2024-05-15T12:45:26.595 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.581+0000 7fa39684b780 20 client.4429.objecter get_session s=0x564ada8d10d0 osd=6 4 2024-05-15T12:45:26.595 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.581+0000 7fa39684b780 15 client.4429.objecter _session_op_assign 6 15 2024-05-15T12:45:26.595 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.581+0000 7fa39684b780 15 client.4429.objecter _send_op 15 to 6.1 on osd.6 2024-05-15T12:45:26.595 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.581+0000 7fa39684b780 1 -- 172.21.15.117:0/4241893909 --> [v2:172.21.15.178:6800/1383036222,v1:172.21.15.178:6801/1383036222] -- osd_op(unknown.0.0:15 6.1 6:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47) v9 -- 0x564ada8b28f0 con 0x564ada976cf0 2024-05-15T12:45:26.595 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.581+0000 7fa39684b780 20 client.4429.objecter put_session s=0x564ada8d10d0 osd=6 5 2024-05-15T12:45:26.596 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.581+0000 7fa39684b780 5 client.4429.objecter 2 in flight 2024-05-15T12:45:26.596 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.605+0000 7fa3942b2640 1 -- 172.21.15.117:0/4241893909 <== osd.6 v2:172.21.15.178:6800/1383036222 16 ==== osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7fa38c001820 con 0x564ada976cf0 2024-05-15T12:45:26.596 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.605+0000 7fa3942b2640 10 client.4429.objecter ms_dispatch 0x564ada7af9d0 osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2024-05-15T12:45:26.596 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.605+0000 7fa3942b2640 10 client.4429.objecter in handle_osd_op_reply 2024-05-15T12:45:26.596 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.605+0000 7fa3942b2640 7 client.4429.objecter handle_osd_op_reply 14 ondisk uv 4 in 6.1 attempt 0 2024-05-15T12:45:26.596 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.605+0000 7fa3942b2640 10 client.4429.objecter op 0 rval 0 len 5 2024-05-15T12:45:26.596 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.605+0000 7fa3942b2640 15 client.4429.objecter handle_osd_op_reply completed tid 14 2024-05-15T12:45:26.596 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.605+0000 7fa3942b2640 15 client.4429.objecter _finish_op 14 2024-05-15T12:45:26.596 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.605+0000 7fa3942b2640 20 client.4429.objecter put_session s=0x564ada8d10d0 osd=6 5 2024-05-15T12:45:26.596 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.605+0000 7fa3942b2640 15 client.4429.objecter _session_op_remove 6 14 2024-05-15T12:45:26.596 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.605+0000 7fa3942b2640 5 client.4429.objecter 1 in flight 2024-05-15T12:45:26.596 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.605+0000 7fa3942b2640 1 -- 172.21.15.117:0/4241893909 <== osd.6 v2:172.21.15.178:6800/1383036222 17 ==== osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7fa38c001820 con 0x564ada976cf0 2024-05-15T12:45:26.596 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.605+0000 7fa3942b2640 10 client.4429.objecter ms_dispatch 0x564ada7af9d0 osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-05-15T12:45:26.596 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.605+0000 7fa3942b2640 10 client.4429.objecter in handle_osd_op_reply 2024-05-15T12:45:26.597 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.605+0000 7fa3942b2640 7 client.4429.objecter handle_osd_op_reply 15 ondisk uv 0 in 6.1 attempt 0 2024-05-15T12:45:26.597 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.605+0000 7fa3942b2640 10 client.4429.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-05-15T12:45:26.597 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.605+0000 7fa3942b2640 10 client.4429.objecter op 0 rval 0 len 0 2024-05-15T12:45:26.597 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.605+0000 7fa3942b2640 15 client.4429.objecter handle_osd_op_reply completed tid 15 2024-05-15T12:45:26.597 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.605+0000 7fa3942b2640 15 client.4429.objecter _finish_op 15 2024-05-15T12:45:26.609 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.605+0000 7fa3942b2640 20 client.4429.objecter put_session s=0x564ada8d10d0 osd=6 4 2024-05-15T12:45:26.609 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.605+0000 7fa3942b2640 15 client.4429.objecter _session_op_remove 6 15 2024-05-15T12:45:26.609 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.605+0000 7fa3942b2640 5 client.4429.objecter 0 in flight 2024-05-15T12:45:26.609 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (24 ms) 2024-05-15T12:45:26.609 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2024-05-15T12:45:26.610 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.605+0000 7fa39684b780 10 client.4429.objecter _op_submit op 0x564ada8b4fb0 2024-05-15T12:45:26.610 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.605+0000 7fa39684b780 20 client.4429.objecter _calc_target epoch 47 base exist @6 precalc_pgid 0 pgid 0.0 is_read 2024-05-15T12:45:26.610 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.605+0000 7fa39684b780 20 client.4429.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-05-15T12:45:26.610 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.605+0000 7fa39684b780 10 client.4429.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [6,3] primary 6 2024-05-15T12:45:26.610 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.605+0000 7fa39684b780 20 client.4429.objecter _get_session s=0x564ada8d10d0 osd=6 3 2024-05-15T12:45:26.610 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.605+0000 7fa39684b780 10 client.4429.objecter _op_submit oid exist '@6' '@6' [read 0~0] tid 16 osd.6 2024-05-15T12:45:26.610 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.605+0000 7fa39684b780 20 client.4429.objecter get_session s=0x564ada8d10d0 osd=6 3 2024-05-15T12:45:26.610 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.605+0000 7fa39684b780 15 client.4429.objecter _session_op_assign 6 16 2024-05-15T12:45:26.610 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.605+0000 7fa39684b780 15 client.4429.objecter _send_op 16 to 6.1 on osd.6 2024-05-15T12:45:26.610 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.605+0000 7fa39684b780 1 -- 172.21.15.117:0/4241893909 --> [v2:172.21.15.178:6800/1383036222,v1:172.21.15.178:6801/1383036222] -- osd_op(unknown.0.0:16 6.1 6:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47) v9 -- 0x564ada976440 con 0x564ada976cf0 2024-05-15T12:45:26.610 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.605+0000 7fa39684b780 20 client.4429.objecter put_session s=0x564ada8d10d0 osd=6 4 2024-05-15T12:45:26.610 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.605+0000 7fa39684b780 5 client.4429.objecter 1 in flight 2024-05-15T12:45:26.610 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.605+0000 7fa39684b780 10 client.4429.objecter _op_submit op 0x564ada8b2ab0 2024-05-15T12:45:26.610 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.605+0000 7fa39684b780 20 client.4429.objecter _calc_target epoch 47 base noexist @6 precalc_pgid 0 pgid 0.0 is_read 2024-05-15T12:45:26.611 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.605+0000 7fa39684b780 20 client.4429.objecter _calc_target target noexist @6 -> pgid 6.a7bbaa1 2024-05-15T12:45:26.611 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.605+0000 7fa39684b780 10 client.4429.objecter _calc_target raw pgid 6.a7bbaa1 -> actual 6.1 acting [6,3] primary 6 2024-05-15T12:45:26.611 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.605+0000 7fa39684b780 20 client.4429.objecter _get_session s=0x564ada8d10d0 osd=6 4 2024-05-15T12:45:26.611 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.605+0000 7fa39684b780 10 client.4429.objecter _op_submit oid noexist '@6' '@6' [read 0~0] tid 17 osd.6 2024-05-15T12:45:26.611 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.605+0000 7fa39684b780 20 client.4429.objecter get_session s=0x564ada8d10d0 osd=6 4 2024-05-15T12:45:26.611 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.605+0000 7fa39684b780 15 client.4429.objecter _session_op_assign 6 17 2024-05-15T12:45:26.611 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.605+0000 7fa39684b780 15 client.4429.objecter _send_op 17 to 6.1 on osd.6 2024-05-15T12:45:26.611 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.605+0000 7fa39684b780 1 -- 172.21.15.117:0/4241893909 --> [v2:172.21.15.178:6800/1383036222,v1:172.21.15.178:6801/1383036222] -- osd_op(unknown.0.0:17 6.1 6:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47) v9 -- 0x564ada8b2f60 con 0x564ada976cf0 2024-05-15T12:45:26.611 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.605+0000 7fa39684b780 20 client.4429.objecter put_session s=0x564ada8d10d0 osd=6 5 2024-05-15T12:45:26.611 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.605+0000 7fa39684b780 5 client.4429.objecter 2 in flight 2024-05-15T12:45:26.611 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.619+0000 7fa3942b2640 1 -- 172.21.15.117:0/4241893909 <== osd.6 v2:172.21.15.178:6800/1383036222 18 ==== osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7fa38c001820 con 0x564ada976cf0 2024-05-15T12:45:26.611 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.619+0000 7fa3942b2640 10 client.4429.objecter ms_dispatch 0x564ada7af9d0 osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2024-05-15T12:45:26.611 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.619+0000 7fa3942b2640 10 client.4429.objecter in handle_osd_op_reply 2024-05-15T12:45:26.611 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.619+0000 7fa3942b2640 7 client.4429.objecter handle_osd_op_reply 16 ondisk uv 4 in 6.1 attempt 0 2024-05-15T12:45:26.612 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.619+0000 7fa3942b2640 10 client.4429.objecter op 0 rval 0 len 5 2024-05-15T12:45:26.612 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.619+0000 7fa3942b2640 15 client.4429.objecter handle_osd_op_reply completed tid 16 2024-05-15T12:45:26.612 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.619+0000 7fa3942b2640 15 client.4429.objecter _finish_op 16 2024-05-15T12:45:26.612 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.619+0000 7fa3942b2640 20 client.4429.objecter put_session s=0x564ada8d10d0 osd=6 5 2024-05-15T12:45:26.612 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.619+0000 7fa3942b2640 15 client.4429.objecter _session_op_remove 6 16 2024-05-15T12:45:26.612 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.619+0000 7fa3942b2640 5 client.4429.objecter 1 in flight 2024-05-15T12:45:26.612 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.619+0000 7fa3942b2640 1 -- 172.21.15.117:0/4241893909 <== osd.6 v2:172.21.15.178:6800/1383036222 19 ==== osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7fa38c001820 con 0x564ada976cf0 2024-05-15T12:45:26.612 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.619+0000 7fa3942b2640 10 client.4429.objecter ms_dispatch 0x564ada7af9d0 osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-05-15T12:45:26.612 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.619+0000 7fa3942b2640 10 client.4429.objecter in handle_osd_op_reply 2024-05-15T12:45:26.612 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.619+0000 7fa3942b2640 7 client.4429.objecter handle_osd_op_reply 17 ondisk uv 0 in 6.1 attempt 0 2024-05-15T12:45:26.612 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.619+0000 7fa3942b2640 10 client.4429.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-05-15T12:45:26.612 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.619+0000 7fa3942b2640 10 client.4429.objecter op 0 rval 0 len 0 2024-05-15T12:45:26.612 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.619+0000 7fa3942b2640 15 client.4429.objecter handle_osd_op_reply completed tid 17 2024-05-15T12:45:26.612 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.619+0000 7fa3942b2640 15 client.4429.objecter _finish_op 17 2024-05-15T12:45:26.612 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.619+0000 7fa3942b2640 20 client.4429.objecter put_session s=0x564ada8d10d0 osd=6 4 2024-05-15T12:45:26.613 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.619+0000 7fa3942b2640 15 client.4429.objecter _session_op_remove 6 17 2024-05-15T12:45:26.613 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.619+0000 7fa3942b2640 5 client.4429.objecter 0 in flight 2024-05-15T12:45:26.613 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (14 ms) 2024-05-15T12:45:26.613 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2024-05-15T12:45:26.613 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.619+0000 7fa39684b780 20 librados: queue_aio_write 0x564ada778830 completion 0x564ada8b5b60 write_seq 4 2024-05-15T12:45:26.613 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.619+0000 7fa39684b780 10 client.4429.objecter _op_submit op 0x564ada8b4fb0 2024-05-15T12:45:26.613 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.619+0000 7fa39684b780 20 client.4429.objecter _calc_target epoch 47 base exist @6 precalc_pgid 0 pgid 0.0 is_write 2024-05-15T12:45:26.613 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.619+0000 7fa39684b780 20 client.4429.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-05-15T12:45:26.613 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.619+0000 7fa39684b780 10 client.4429.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [6,3] primary 6 2024-05-15T12:45:26.613 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.619+0000 7fa39684b780 20 client.4429.objecter _get_session s=0x564ada8d10d0 osd=6 3 2024-05-15T12:45:26.649 INFO:tasks.ceph.osd.4.smithi178.stderr:2024-05-15T12:45:26.647+0000 12a65640 -1 osd.4 47 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4436.0:1 39.6s0 39.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-05-15T12:45:26.681 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.619+0000 7fa39684b780 10 client.4429.objecter _op_submit oid exist '@6' '@6' [writefull 0~5 in=5b] tid 18 osd.6 2024-05-15T12:45:26.681 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.619+0000 7fa39684b780 20 client.4429.objecter get_session s=0x564ada8d10d0 osd=6 3 2024-05-15T12:45:26.682 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.619+0000 7fa39684b780 15 client.4429.objecter _session_op_assign 6 18 2024-05-15T12:45:26.682 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.619+0000 7fa39684b780 15 client.4429.objecter _send_op 18 to 6.1 on osd.6 2024-05-15T12:45:26.682 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.619+0000 7fa39684b780 1 -- 172.21.15.117:0/4241893909 --> [v2:172.21.15.178:6800/1383036222,v1:172.21.15.178:6801/1383036222] -- osd_op(unknown.0.0:18 6.1 6:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47) v9 -- 0x564ada976440 con 0x564ada976cf0 2024-05-15T12:45:26.682 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.619+0000 7fa39684b780 20 client.4429.objecter put_session s=0x564ada8d10d0 osd=6 4 2024-05-15T12:45:26.682 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.619+0000 7fa39684b780 5 client.4429.objecter 1 in flight 2024-05-15T12:45:26.682 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.691+0000 7fa3942b2640 1 -- 172.21.15.117:0/4241893909 <== osd.6 v2:172.21.15.178:6800/1383036222 20 ==== osd_op_reply(18 exist [writefull 0~5] v47'5 uv5 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7fa38c001820 con 0x564ada976cf0 2024-05-15T12:45:26.682 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.691+0000 7fa3942b2640 10 client.4429.objecter ms_dispatch 0x564ada7af9d0 osd_op_reply(18 exist [writefull 0~5] v47'5 uv5 ondisk = 0) v8 2024-05-15T12:45:26.682 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.691+0000 7fa3942b2640 10 client.4429.objecter in handle_osd_op_reply 2024-05-15T12:45:26.682 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.691+0000 7fa3942b2640 7 client.4429.objecter handle_osd_op_reply 18 ondisk uv 5 in 6.1 attempt 0 2024-05-15T12:45:26.682 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.691+0000 7fa3942b2640 10 client.4429.objecter op 0 rval 0 len 0 2024-05-15T12:45:26.682 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.691+0000 7fa3942b2640 15 client.4429.objecter handle_osd_op_reply completed tid 18 2024-05-15T12:45:26.683 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.691+0000 7fa3942b2640 15 client.4429.objecter _finish_op 18 2024-05-15T12:45:26.683 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.691+0000 7fa3942b2640 20 client.4429.objecter put_session s=0x564ada8d10d0 osd=6 4 2024-05-15T12:45:26.683 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.691+0000 7fa3942b2640 15 client.4429.objecter _session_op_remove 6 18 2024-05-15T12:45:26.683 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.691+0000 7fa3942b2640 5 client.4429.objecter 0 in flight 2024-05-15T12:45:26.683 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.691+0000 7fa3942b2640 20 librados: complete_aio_write 0x564ada8b5b60 2024-05-15T12:45:26.683 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (72 ms) 2024-05-15T12:45:26.683 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2024-05-15T12:45:26.683 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.691+0000 7fa39684b780 20 librados: queue_aio_write 0x564ada778830 completion 0x564ada976330 write_seq 5 2024-05-15T12:45:26.683 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.691+0000 7fa39684b780 10 client.4429.objecter _op_submit op 0x564ada8b4fb0 2024-05-15T12:45:26.683 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.691+0000 7fa39684b780 20 client.4429.objecter _calc_target epoch 47 base exist @6 precalc_pgid 0 pgid 0.0 is_write 2024-05-15T12:45:26.683 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.691+0000 7fa39684b780 20 client.4429.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-05-15T12:45:26.683 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.691+0000 7fa39684b780 10 client.4429.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [6,3] primary 6 2024-05-15T12:45:26.684 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.691+0000 7fa39684b780 20 client.4429.objecter _get_session s=0x564ada8d10d0 osd=6 3 2024-05-15T12:45:26.684 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.691+0000 7fa39684b780 10 client.4429.objecter _op_submit oid exist '@6' '@6' [writefull 0~5 in=5b] tid 19 osd.6 2024-05-15T12:45:26.684 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.691+0000 7fa39684b780 20 client.4429.objecter get_session s=0x564ada8d10d0 osd=6 3 2024-05-15T12:45:26.684 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.691+0000 7fa39684b780 15 client.4429.objecter _session_op_assign 6 19 2024-05-15T12:45:26.762 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.691+0000 7fa39684b780 15 client.4429.objecter _send_op 19 to 6.1 on osd.6 2024-05-15T12:45:26.763 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.691+0000 7fa39684b780 1 -- 172.21.15.117:0/4241893909 --> [v2:172.21.15.178:6800/1383036222,v1:172.21.15.178:6801/1383036222] -- osd_op(unknown.0.0:19 6.1 6:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47) v9 -- 0x564ada976440 con 0x564ada976cf0 2024-05-15T12:45:26.763 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.691+0000 7fa39684b780 20 client.4429.objecter put_session s=0x564ada8d10d0 osd=6 4 2024-05-15T12:45:26.763 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.691+0000 7fa39684b780 5 client.4429.objecter 1 in flight 2024-05-15T12:45:26.763 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.733+0000 7fa37bfff640 10 monclient: tick 2024-05-15T12:45:26.763 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.733+0000 7fa37bfff640 10 monclient: _check_auth_tickets 2024-05-15T12:45:26.763 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.733+0000 7fa37bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:45:26.763 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.772+0000 7fa3942b2640 1 -- 172.21.15.117:0/4241893909 <== osd.6 v2:172.21.15.178:6800/1383036222 21 ==== osd_op_reply(19 exist [writefull 0~5] v47'6 uv6 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7fa38c001820 con 0x564ada976cf0 2024-05-15T12:45:26.763 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.772+0000 7fa3942b2640 10 client.4429.objecter ms_dispatch 0x564ada7af9d0 osd_op_reply(19 exist [writefull 0~5] v47'6 uv6 ondisk = 0) v8 2024-05-15T12:45:26.763 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.772+0000 7fa3942b2640 10 client.4429.objecter in handle_osd_op_reply 2024-05-15T12:45:26.763 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.772+0000 7fa3942b2640 7 client.4429.objecter handle_osd_op_reply 19 ondisk uv 6 in 6.1 attempt 0 2024-05-15T12:45:26.763 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.772+0000 7fa3942b2640 10 client.4429.objecter op 0 rval 0 len 0 2024-05-15T12:45:26.763 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.772+0000 7fa3942b2640 15 client.4429.objecter handle_osd_op_reply completed tid 19 2024-05-15T12:45:26.763 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.772+0000 7fa3942b2640 15 client.4429.objecter _finish_op 19 2024-05-15T12:45:26.763 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.772+0000 7fa3942b2640 20 client.4429.objecter put_session s=0x564ada8d10d0 osd=6 4 2024-05-15T12:45:26.763 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.772+0000 7fa3942b2640 15 client.4429.objecter _session_op_remove 6 19 2024-05-15T12:45:26.764 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.772+0000 7fa3942b2640 5 client.4429.objecter 0 in flight 2024-05-15T12:45:26.764 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.772+0000 7fa3942b2640 20 librados: complete_aio_write 0x564ada976330 2024-05-15T12:45:26.764 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (81 ms) 2024-05-15T12:45:26.764 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2024-05-15T12:45:26.764 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.772+0000 7fa39684b780 20 librados: queue_aio_write 0x564ada778830 completion 0x564ada976330 write_seq 6 2024-05-15T12:45:26.764 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.772+0000 7fa39684b780 10 client.4429.objecter _op_submit op 0x564ada8b4fb0 2024-05-15T12:45:26.764 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.772+0000 7fa39684b780 20 client.4429.objecter _calc_target epoch 47 base exist @6 precalc_pgid 0 pgid 0.0 is_write 2024-05-15T12:45:26.764 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.772+0000 7fa39684b780 20 client.4429.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-05-15T12:45:26.764 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.772+0000 7fa39684b780 10 client.4429.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [6,3] primary 6 2024-05-15T12:45:26.764 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.772+0000 7fa39684b780 20 client.4429.objecter _get_session s=0x564ada8d10d0 osd=6 3 2024-05-15T12:45:26.764 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.772+0000 7fa39684b780 10 client.4429.objecter _op_submit oid exist '@6' '@6' [writefull 0~5 in=5b] tid 20 osd.6 2024-05-15T12:45:26.764 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.772+0000 7fa39684b780 20 client.4429.objecter get_session s=0x564ada8d10d0 osd=6 3 2024-05-15T12:45:26.764 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.772+0000 7fa39684b780 15 client.4429.objecter _session_op_assign 6 20 2024-05-15T12:45:27.587 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.772+0000 7fa39684b780 15 client.4429.objecter _send_op 20 to 6.1 on osd.6 2024-05-15T12:45:27.587 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.772+0000 7fa39684b780 1 -- 172.21.15.117:0/4241893909 --> [v2:172.21.15.178:6800/1383036222,v1:172.21.15.178:6801/1383036222] -- osd_op(unknown.0.0:20 6.1 6:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47) v9 -- 0x564ada976440 con 0x564ada976cf0 2024-05-15T12:45:27.588 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.772+0000 7fa39684b780 20 client.4429.objecter put_session s=0x564ada8d10d0 osd=6 4 2024-05-15T12:45:27.588 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:26.772+0000 7fa39684b780 5 client.4429.objecter 1 in flight 2024-05-15T12:45:27.588 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:27.596+0000 7fa3942b2640 1 -- 172.21.15.117:0/4241893909 <== osd.6 v2:172.21.15.178:6800/1383036222 22 ==== osd_op_reply(20 exist [writefull 0~5] v47'7 uv7 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7fa38c001820 con 0x564ada976cf0 2024-05-15T12:45:27.588 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:27.596+0000 7fa3942b2640 10 client.4429.objecter ms_dispatch 0x564ada7af9d0 osd_op_reply(20 exist [writefull 0~5] v47'7 uv7 ondisk = 0) v8 2024-05-15T12:45:27.588 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:27.596+0000 7fa3942b2640 10 client.4429.objecter in handle_osd_op_reply 2024-05-15T12:45:27.588 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:27.596+0000 7fa3942b2640 7 client.4429.objecter handle_osd_op_reply 20 ondisk uv 7 in 6.1 attempt 0 2024-05-15T12:45:27.588 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:27.596+0000 7fa3942b2640 10 client.4429.objecter op 0 rval 0 len 0 2024-05-15T12:45:27.588 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:27.596+0000 7fa3942b2640 15 client.4429.objecter handle_osd_op_reply completed tid 20 2024-05-15T12:45:27.588 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:27.596+0000 7fa3942b2640 15 client.4429.objecter _finish_op 20 2024-05-15T12:45:27.588 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:27.596+0000 7fa3942b2640 20 client.4429.objecter put_session s=0x564ada8d10d0 osd=6 4 2024-05-15T12:45:27.588 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:27.596+0000 7fa3942b2640 15 client.4429.objecter _session_op_remove 6 20 2024-05-15T12:45:27.588 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:27.596+0000 7fa3942b2640 5 client.4429.objecter 0 in flight 2024-05-15T12:45:27.588 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:27.596+0000 7fa3942b2640 20 librados: complete_aio_write 0x564ada976330 2024-05-15T12:45:27.589 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (825 ms) 2024-05-15T12:45:27.589 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:27.597+0000 7fa39684b780 10 librados: watch_flush enter 2024-05-15T12:45:27.589 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:27.597+0000 7fa39684b780 10 librados: watch_flush exit 2024-05-15T12:45:27.589 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:27.597+0000 7fa39684b780 10 client.4429.objecter close_session for osd.6 2024-05-15T12:45:27.589 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:27.597+0000 7fa39684b780 1 -- 172.21.15.117:0/4241893909 >> [v2:172.21.15.178:6800/1383036222,v1:172.21.15.178:6801/1383036222] conn(0x564ada976cf0 msgr2=0x564ada8b0880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:45:27.589 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:27.597+0000 7fa39684b780 1 --2- 172.21.15.117:0/4241893909 >> [v2:172.21.15.178:6800/1383036222,v1:172.21.15.178:6801/1383036222] conn(0x564ada976cf0 0x564ada8b0880 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:45:27.589 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:27.597+0000 7fa39684b780 20 client.4429.objecter put_session s=0x564ada8d10d0 osd=6 1 2024-05-15T12:45:27.589 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:27.597+0000 7fa39684b780 20 client.4429.objecter shutdown clearing up homeless session... 2024-05-15T12:45:27.589 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:27.597+0000 7fa39684b780 10 client.4429.objecter successfully canceled tick 2024-05-15T12:45:27.589 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:27.597+0000 7fa39684b780 1 -- 172.21.15.117:0/4241893909 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fa38006b0d0 msgr2=0x7fa38006d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:45:27.589 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:27.597+0000 7fa39684b780 1 --2- 172.21.15.117:0/4241893909 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fa38006b0d0 0x7fa38006d590 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa384054000 tx=0x7fa38404c070 comp rx=0 tx=0).stop 2024-05-15T12:45:27.591 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:27.597+0000 7fa39684b780 10 monclient: shutdown 2024-05-15T12:45:27.592 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:27.597+0000 7fa39684b780 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:45:27.592 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:27.597+0000 7fa39684b780 1 -- 172.21.15.117:0/4241893909 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564ada8d4330 msgr2=0x564ada808470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:45:27.592 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:27.597+0000 7fa39684b780 1 --2- 172.21.15.117:0/4241893909 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564ada8d4330 0x564ada808470 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa388000c00 tx=0x7fa388003020 comp rx=0 tx=0).stop 2024-05-15T12:45:27.592 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:27.597+0000 7fa3942b2640 1 -- 172.21.15.117:0/4241893909 reap_dead start 2024-05-15T12:45:27.592 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:27.597+0000 7fa39684b780 1 -- 172.21.15.117:0/4241893909 shutdown_connections 2024-05-15T12:45:27.592 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:27.597+0000 7fa39684b780 1 -- 172.21.15.117:0/4241893909 >> 172.21.15.117:0/4241893909 conn(0x564ada843dc0 msgr2=0x564ada8d2170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:45:27.592 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:27.597+0000 7fa39684b780 1 -- 172.21.15.117:0/4241893909 shutdown_connections 2024-05-15T12:45:27.593 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:27.597+0000 7fa39684b780 1 -- 172.21.15.117:0/4241893909 wait complete. 2024-05-15T12:45:27.593 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:27.597+0000 7fa39684b780 1 librados: shutdown 2024-05-15T12:45:27.593 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [----------] 12 tests from AsioRados (19461 ms total) 2024-05-15T12:45:27.593 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-05-15T12:45:27.593 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [----------] Global test environment tear-down 2024-05-15T12:45:27.593 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (41916 ms total) 2024-05-15T12:45:27.593 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [ PASSED ] 12 tests. 2024-05-15T12:45:28.269 INFO:tasks.ceph.osd.4.smithi178.stderr:2024-05-15T12:45:28.081+0000 12a65640 -1 osd.4 48 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4436.0:1 39.6s0 39:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-05-15T12:45:36.214 INFO:tasks.workunit.client.0.smithi117.stdout: api_cls_remote_reads: [==========] Running 1 test from 1 test suite. 2024-05-15T12:45:36.214 INFO:tasks.workunit.client.0.smithi117.stdout: api_cls_remote_reads: [----------] Global test environment set-up. 2024-05-15T12:45:36.215 INFO:tasks.workunit.client.0.smithi117.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads 2024-05-15T12:45:36.215 INFO:tasks.workunit.client.0.smithi117.stdout: api_cls_remote_reads: [ RUN ] ClsTestRemoteReads.TestGather 2024-05-15T12:45:36.215 INFO:tasks.workunit.client.0.smithi117.stdout: api_cls_remote_reads: [ OK ] ClsTestRemoteReads.TestGather (50300 ms) 2024-05-15T12:45:36.215 INFO:tasks.workunit.client.0.smithi117.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads (50300 ms total) 2024-05-15T12:45:36.215 INFO:tasks.workunit.client.0.smithi117.stdout: api_cls_remote_reads: 2024-05-15T12:45:36.215 INFO:tasks.workunit.client.0.smithi117.stdout: api_cls_remote_reads: [----------] Global test environment tear-down 2024-05-15T12:45:36.215 INFO:tasks.workunit.client.0.smithi117.stdout: api_cls_remote_reads: [==========] 1 test from 1 test suite ran. (50300 ms total) 2024-05-15T12:45:36.215 INFO:tasks.workunit.client.0.smithi117.stdout: api_cls_remote_reads: [ PASSED ] 1 test. 2024-05-15T12:45:36.616 INFO:tasks.workunit.client.0.smithi117.stdout: handler_error: Running main() from gmock_main.cc 2024-05-15T12:45:36.616 INFO:tasks.workunit.client.0.smithi117.stdout: handler_error: [==========] Running 1 test from 1 test suite. 2024-05-15T12:45:36.616 INFO:tasks.workunit.client.0.smithi117.stdout: handler_error: [----------] Global test environment set-up. 2024-05-15T12:45:36.616 INFO:tasks.workunit.client.0.smithi117.stdout: handler_error: [----------] 1 test from neocls_handler_error 2024-05-15T12:45:36.616 INFO:tasks.workunit.client.0.smithi117.stdout: handler_error: [ RUN ] neocls_handler_error.test_handler_error 2024-05-15T12:45:36.616 INFO:tasks.workunit.client.0.smithi117.stdout: handler_error: [ OK ] neocls_handler_error.test_handler_error (50630 ms) 2024-05-15T12:45:36.616 INFO:tasks.workunit.client.0.smithi117.stdout: handler_error: [----------] 1 test from neocls_handler_error (50630 ms total) 2024-05-15T12:45:36.616 INFO:tasks.workunit.client.0.smithi117.stdout: handler_error: 2024-05-15T12:45:36.616 INFO:tasks.workunit.client.0.smithi117.stdout: handler_error: [----------] Global test environment tear-down 2024-05-15T12:45:36.617 INFO:tasks.workunit.client.0.smithi117.stdout: handler_error: [==========] 1 test from 1 test suite ran. (50630 ms total) 2024-05-15T12:45:36.617 INFO:tasks.workunit.client.0.smithi117.stdout: handler_error: [ PASSED ] 1 test. 2024-05-15T12:45:39.819 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: Running main() from gmock_main.cc 2024-05-15T12:45:39.819 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2024-05-15T12:45:39.819 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: [----------] Global test environment set-up. 2024-05-15T12:45:39.820 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2024-05-15T12:45:39.820 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2024-05-15T12:45:39.820 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (1113 ms) 2024-05-15T12:45:39.820 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2024-05-15T12:45:39.820 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (101 ms) 2024-05-15T12:45:39.820 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2024-05-15T12:45:39.820 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (50970 ms) 2024-05-15T12:45:39.820 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2024-05-15T12:45:39.820 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:30.566543+0000 mon.a [WRN] Health check update: 19 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-15T12:45:39.820 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:30.584270+0000 mon.a [WRN] Health check failed: 3 slow ops, oldest one blocked for 34 sec, daemons [osd.0,osd.2,osd.4] have slow ops. (SLOW_OPS) 2024-05-15T12:45:39.820 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:35.468267+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi117-40272-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-05-15T12:45:39.820 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:36.252548+0000 mon.c [INF] from='client.? 172.21.15.117:0/1059831129' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi117-40984-1"}]: dispatch 2024-05-15T12:45:39.820 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:36.651492+0000 mon.b [INF] from='client.? 172.21.15.117:0/1699290781' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi117-40236-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-05-15T12:45:39.820 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:36.690564+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi117-40984-1"}]: dispatch 2024-05-15T12:45:39.821 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:37.479865+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi117-40236-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-05-15T12:45:39.821 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:37.863203+0000 mon.b [INF] from='client.? 172.21.15.117:0/2179658584' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi117-40389-1"}]: dispatch 2024-05-15T12:45:39.821 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:37.980267+0000 mon.c [INF] from='client.? 172.21.15.117:0/4261581187' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi117-40560-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-05-15T12:45:39.821 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:38.003949+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi117-40389-1"}]: dispatch 2024-05-15T12:45:39.821 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:38.034557+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi117-40560-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-05-15T12:45:39.821 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:38.084788+0000 mon.c [INF] from='client.? 172.21.15.117:0/517788769' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi117-39970-16"}]: dispatch 2024-05-15T12:45:39.821 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:38.070441+0000 mon.b [INF] from='client.? 172.21.15.117:0/1963543561' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-05-15T12:45:39.821 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:38.132934+0000 mon.b [INF] from='client.? 172.21.15.117:0/664207868' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi117-40044-10"}]: dispatch 2024-05-15T12:45:39.821 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:38.193286+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi117-39970-16"}]: dispatch 2024-05-15T12:45:39.821 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:38.212432+0000 mon.a [WRN] Health check update: 15 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-15T12:45:39.821 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:38.213137+0000 mon.a [INF] Health check cleared: SLOW_OPS (was: 2 slow ops, oldest one blocked for 33 sec, osd.4 has slow ops) 2024-05-15T12:45:39.821 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:38.227163+0000 client.admin [INF] onexx 2024-05-15T12:45:39.821 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:38.231029+0000 mon.c [INF] from='client.? 172.21.15.117:0/517788769' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi117-39970-16"}]: dispatch 2024-05-15T12:45:39.822 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:38.238862+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-05-15T12:45:39.822 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:38.248348+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi117-40044-10"}]: dispatch 2024-05-15T12:45:39.822 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:38.289451+0000 mon.b [INF] from='client.? 172.21.15.117:0/664207868' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi117-40044-10"}]: dispatch 2024-05-15T12:45:39.822 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:38.458138+0000 mon.b [INF] from='client.? 172.21.15.117:0/3939347145' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi117-39980-22"}]: dispatch 2024-05-15T12:45:39.822 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:38.473362+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi117-39970-16"}]: dispatch 2024-05-15T12:45:39.822 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:38.491290+0000 mon.c [INF] from='client.? 172.21.15.117:0/2397832852' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi117-40247-7"}]: dispatch 2024-05-15T12:45:39.822 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:38.543070+0000 mon.c [INF] from='client.? 172.21.15.117:0/3651275886' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi117-40420-7"}]: dispatch 2024-05-15T12:45:39.822 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:38.571221+0000 mon.c [INF] from='client.? 172.21.15.117:0/517788769' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi117-39970-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-05-15T12:45:39.822 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:38.583181+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi117-40044-10"}]: dispatch 2024-05-15T12:45:39.822 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:38.590140+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi117-39980-22"}]: dispatch 2024-05-15T12:45:39.822 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:38.813104+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi117-40247-7"}]: dispatch 2024-05-15T12:45:39.822 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:38.821011+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi117-40420-7"}]: dispatch 2024-05-15T12:45:39.822 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:38.825370+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi117-39970-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-05-15T12:45:43.018 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:38.842409+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi117-40984-1"}]': finished 2024-05-15T12:45:43.018 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:38.850970+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi117-40236-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-05-15T12:45:43.018 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:38.852854+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi117-40389-1"}]': finished 2024-05-15T12:45:43.018 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:38.855304+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi117-40560-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-05-15T12:45:43.018 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:39.111756+0000 mon.c [INF] from='client.? 172.21.15.117:0/1059831129' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi117-40984-1"}]: dispatch 2024-05-15T12:45:43.018 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:39.112379+0000 mon.b [INF] from='client.? 172.21.15.117:0/2179658584' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi117-40389-1"}]: dispatch 2024-05-15T12:45:43.019 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:39.138471+0000 mon.c [INF] from='client.? 172.21.15.117:0/2397832852' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi117-40247-7"}]: dispatch 2024-05-15T12:45:43.019 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:39.142747+0000 mon.b [INF] from='client.? 172.21.15.117:0/3939347145' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi117-39980-22"}]: dispatch 2024-05-15T12:45:43.019 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:39.155214+0000 mon.b [INF] from='client.? 172.21.15.117:0/664207868' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi117-40044-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-05-15T12:45:43.019 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:40.759376+0000 mon.c [INF] from='client.? 172.21.15.117:0/2434913738' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi117-40984-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-05-15T12:45:43.019 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:40.769594+0000 mon.b [INF] from='client.? 172.21.15.117:0/664207868' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi117-40044-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi117-40044-10"}]: dispatch 2024-05-15T12:45:43.019 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:40.779788+0000 mon.b [INF] from='client.? 172.21.15.117:0/2801646039' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi117-40102-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-05-15T12:45:43.019 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:40.789403+0000 mon.b [INF] from='client.? 172.21.15.117:0/1699290781' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi117-40236-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-05-15T12:45:43.019 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:40.802644+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-05-15T12:45:43.019 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:40.959666+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi117-40984-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-05-15T12:45:43.260 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:40.987888+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi117-40044-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi117-40044-10"}]: dispatch 2024-05-15T12:45:43.260 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:41.005857+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi117-40102-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-05-15T12:45:43.260 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:41.013509+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi117-40236-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-05-15T12:45:43.260 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:41.432075+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi117-40420-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-05-15T12:45:43.260 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:41.435364+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi117-40247-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-05-15T12:45:43.260 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:41.437939+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi117-39980-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-05-15T12:45:43.260 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:41.440409+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi117-40984-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-05-15T12:45:43.260 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:41.443016+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi117-40102-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-05-15T12:45:43.260 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:41.445434+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi117-40236-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-05-15T12:45:43.260 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:41.779122+0000 mon.b [INF] from='client.? 172.21.15.117:0/3939347145' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi117-39980-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi117-39980-22"}]: dispatch 2024-05-15T12:45:43.261 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:41.786323+0000 mon.c [INF] from='client.? 172.21.15.117:0/2397832852' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi117-40247-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi117-40247-7"}]: dispatch 2024-05-15T12:45:43.261 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:41.797892+0000 mon.c [INF] from='client.? 172.21.15.117:0/2434913738' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi117-40984-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi117-40984-2"}]: dispatch 2024-05-15T12:45:43.261 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:41.811370+0000 mon.b [INF] from='client.? 172.21.15.117:0/1920837811' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi117-40099-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-05-15T12:45:43.261 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:41.812520+0000 mon.c [INF] from='client.? 172.21.15.117:0/3651275886' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi117-40420-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi117-40420-7"}]: dispatch 2024-05-15T12:45:43.261 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:41.823489+0000 mon.b [INF] from='client.? 172.21.15.117:0/1262203275' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi117-40389-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-05-15T12:45:43.261 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:41.828060+0000 mon.b [INF] from='client.? 172.21.15.117:0/2801646039' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi117-40102-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi117-40102-10"}]: dispatch 2024-05-15T12:45:43.261 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:41.850262+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi117-40247-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi117-40247-7"}]: dispatch 2024-05-15T12:45:43.261 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:41.857347+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi117-39980-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi117-39980-22"}]: dispatch 2024-05-15T12:45:43.261 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:41.865830+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi117-40984-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi117-40984-2"}]: dispatch 2024-05-15T12:45:43.261 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:41.872489+0000 mon.b [INF] from='client.? 172.21.15.117:0/492924831' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi117-40272-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-05-15T12:45:43.261 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:41.873527+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi117-40420-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi117-40420-7"}]: dispatch 2024-05-15T12:45:43.261 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:41.903714+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi117-40099-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-05-15T12:45:43.261 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:42.096833+0000 mon.b [INF] from='client.? 172.21.15.117:0/1699290781' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi117-40236-4", "tierpool": "test-rados-api-smithi117-40236-4-cache"}]: dispatch 2024-05-15T12:45:43.261 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:42.118427+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi117-40389-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-05-15T12:45:43.262 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:42.138447+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi117-40102-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi117-40102-10"}]: dispatch 2024-05-15T12:45:43.262 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:42.148738+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi117-40272-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-05-15T12:45:43.262 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:42.255436+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi117-40236-4", "tierpool": "test-rados-api-smithi117-40236-4-cache"}]: dispatch 2024-05-15T12:45:45.542 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:03:52.713 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:45.544 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:03:52.722 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:45.547 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:03:52.719 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:45.547 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:03:52.726 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:45.614 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:03:52.792 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:45.618 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:03:52.797 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:45.691 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:03:52.870 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:45.728 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:03:52.899 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:45.762 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:03:52.941 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:46.525 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:42.689778+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi117-39970-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi117-39970-16"}]': finished 2024-05-15T12:45:46.525 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:42.704035+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi117-40044-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi117-40044-10"}]': finished 2024-05-15T12:45:46.525 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:42.719624+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi117-40099-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-05-15T12:45:46.525 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:42.721653+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi117-40389-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-05-15T12:45:46.525 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:42.728198+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi117-40272-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-05-15T12:45:46.526 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:42.733695+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi117-40236-4", "tierpool": "test-rados-api-smithi117-40236-4-cache"}]': finished 2024-05-15T12:45:46.526 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:43.014960+0000 mon.b [INF] from='client.? 172.21.15.117:0/1963543561' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-05-15T12:45:46.526 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:43.050646+0000 mon.b [INF] from='client.? 172.21.15.117:0/1699290781' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi117-40236-4", "overlaypool": "test-rados-api-smithi117-40236-4-cache"}]: dispatch 2024-05-15T12:45:46.526 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-05-15T12:45:43.093895+0000 client.admin [INF] threexx 2024-05-15T12:45:46.526 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (8560 ms) 2024-05-15T12:45:46.526 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (60744 ms total) 2024-05-15T12:45:46.526 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: 2024-05-15T12:45:46.526 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: [----------] Global test environment tear-down 2024-05-15T12:45:46.526 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (60744 ms total) 2024-05-15T12:45:46.526 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: [ PASSED ] 4 tests. 2024-05-15T12:46:05.905 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:04:13.083 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:05.942 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:04:13.120 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:05.983 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:04:13.162 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:05.985 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:04:13.164 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:06.035 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:04:13.207 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:06.048 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:04:13.219 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:06.149 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:04:13.327 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:06.206 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:04:13.378 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:06.471 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:04:13.649 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:09.250 INFO:tasks.workunit.client.0.smithi117.stdout: api_pool: Running main() from gmock_main.cc 2024-05-15T12:46:09.251 INFO:tasks.workunit.client.0.smithi117.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2024-05-15T12:46:09.251 INFO:tasks.workunit.client.0.smithi117.stdout: api_pool: [----------] Global test environment set-up. 2024-05-15T12:46:09.251 INFO:tasks.workunit.client.0.smithi117.stdout: api_pool: [----------] 9 tests from LibRadosPools 2024-05-15T12:46:09.251 INFO:tasks.workunit.client.0.smithi117.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2024-05-15T12:46:09.251 INFO:tasks.workunit.client.0.smithi117.stdout: api_pool: [ OK ] LibRadosPools.PoolList (7895 ms) 2024-05-15T12:46:09.251 INFO:tasks.workunit.client.0.smithi117.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2024-05-15T12:46:09.251 INFO:tasks.workunit.client.0.smithi117.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (6919 ms) 2024-05-15T12:46:09.251 INFO:tasks.workunit.client.0.smithi117.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2024-05-15T12:46:09.251 INFO:tasks.workunit.client.0.smithi117.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (5156 ms) 2024-05-15T12:46:09.251 INFO:tasks.workunit.client.0.smithi117.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2024-05-15T12:46:09.251 INFO:tasks.workunit.client.0.smithi117.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (11019 ms) 2024-05-15T12:46:09.251 INFO:tasks.workunit.client.0.smithi117.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2024-05-15T12:46:09.251 INFO:tasks.workunit.client.0.smithi117.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (22481 ms) 2024-05-15T12:46:09.251 INFO:tasks.workunit.client.0.smithi117.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2024-05-15T12:46:09.252 INFO:tasks.workunit.client.0.smithi117.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (7505 ms) 2024-05-15T12:46:09.252 INFO:tasks.workunit.client.0.smithi117.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2024-05-15T12:46:09.252 INFO:tasks.workunit.client.0.smithi117.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (5943 ms) 2024-05-15T12:46:09.252 INFO:tasks.workunit.client.0.smithi117.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2024-05-15T12:46:09.252 INFO:tasks.workunit.client.0.smithi117.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (5706 ms) 2024-05-15T12:46:09.252 INFO:tasks.workunit.client.0.smithi117.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2024-05-15T12:46:09.252 INFO:tasks.workunit.client.0.smithi117.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (10904 ms) 2024-05-15T12:46:09.252 INFO:tasks.workunit.client.0.smithi117.stdout: api_pool: [----------] 9 tests from LibRadosPools (83528 ms total) 2024-05-15T12:46:09.252 INFO:tasks.workunit.client.0.smithi117.stdout: api_pool: 2024-05-15T12:46:09.252 INFO:tasks.workunit.client.0.smithi117.stdout: api_pool: [----------] Global test environment tear-down 2024-05-15T12:46:09.252 INFO:tasks.workunit.client.0.smithi117.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (83528 ms total) 2024-05-15T12:46:09.252 INFO:tasks.workunit.client.0.smithi117.stdout: api_pool: [ PASSED ] 9 tests. 2024-05-15T12:46:09.806 INFO:tasks.ceph.osd.0.smithi117.stderr:2024-05-15T12:46:09.809+0000 12a65640 -1 osd.0 75 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4561.0:95 34.9 34:91fb224f:test-rados-api-smithi117-40597-15::testobj:head [omap-set-vals in=62b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50) 2024-05-15T12:46:11.036 INFO:tasks.ceph.osd.0.smithi117.stderr:2024-05-15T12:46:10.999+0000 12a65640 -1 osd.0 76 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4561.0:95 34.9 34:91fb224f:test-rados-api-smithi117-40597-15::testobj:head [omap-set-vals in=62b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50) 2024-05-15T12:46:11.929 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-05-15T12:46:11.867+0000 12a65640 -1 osd.2 77 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4735.0:2 66.6 66:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51) 2024-05-15T12:46:12.143 INFO:tasks.ceph.osd.0.smithi117.stderr:2024-05-15T12:46:12.063+0000 12a65640 -1 osd.0 77 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4561.0:95 34.9 34:91fb224f:test-rados-api-smithi117-40597-15::testobj:head [omap-set-vals in=62b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50) 2024-05-15T12:46:13.094 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-05-15T12:46:12.984+0000 12a65640 -1 osd.2 78 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4735.0:2 66.6 66:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51) 2024-05-15T12:46:15.292 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-05-15T12:46:15.249+0000 12a65640 -1 osd.2 80 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4735.0:2 66.6 66:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51) 2024-05-15T12:46:16.529 INFO:tasks.ceph.osd.3.smithi117.stderr:2024-05-15T12:46:16.529+0000 ea5d640 -1 osd.3 81 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4700.0:2 71.0 71.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e53) 2024-05-15T12:46:17.873 INFO:tasks.ceph.osd.3.smithi117.stderr:2024-05-15T12:46:17.872+0000 ea5d640 -1 osd.3 83 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4700.0:2 71.0 71.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e53) 2024-05-15T12:46:19.442 INFO:tasks.ceph.osd.3.smithi117.stderr:2024-05-15T12:46:19.342+0000 ea5d640 -1 osd.3 84 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4700.0:2 71.0 71.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e53) 2024-05-15T12:46:20.549 INFO:tasks.ceph.osd.3.smithi117.stderr:2024-05-15T12:46:20.504+0000 ea5d640 -1 osd.3 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4700.0:2 71.0 71.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e53) 2024-05-15T12:46:21.985 INFO:tasks.ceph.osd.1.smithi117.stderr:2024-05-15T12:46:21.792+0000 12a65640 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4706.0:1 79.3s0 79.be9754b3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e56) 2024-05-15T12:46:22.026 INFO:tasks.ceph.osd.3.smithi117.stderr:2024-05-15T12:46:22.024+0000 ea5d640 -1 osd.3 87 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4700.0:2 71.0 71:0384ab8e:::oid:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53) 2024-05-15T12:46:23.419 INFO:tasks.ceph.osd.1.smithi117.stderr:2024-05-15T12:46:23.425+0000 12a65640 -1 osd.1 86 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4706.0:1 79.3s0 79.be9754b3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e56) 2024-05-15T12:46:23.423 INFO:tasks.ceph.osd.3.smithi117.stderr:2024-05-15T12:46:23.393+0000 ea5d640 -1 osd.3 88 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4700.0:2 71.0 71:0384ab8e:::oid:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53) 2024-05-15T12:46:23.445 INFO:tasks.ceph.osd.7.smithi178.stderr:2024-05-15T12:46:23.445+0000 ea5d640 -1 osd.7 89 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4658.0:2 65.15 65.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e52) 2024-05-15T12:46:24.535 INFO:tasks.ceph.osd.7.smithi178.stderr:2024-05-15T12:46:24.500+0000 ea5d640 -1 osd.7 90 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4691.0:1 80.0s0 80.fc7118c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e56) 2024-05-15T12:46:24.743 INFO:tasks.ceph.osd.3.smithi117.stderr:2024-05-15T12:46:24.645+0000 ea5d640 -1 osd.3 89 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4700.0:2 71.0 71:0384ab8e:::oid:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53) 2024-05-15T12:46:24.802 INFO:tasks.ceph.osd.1.smithi117.stderr:2024-05-15T12:46:24.619+0000 12a65640 -1 osd.1 88 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4706.0:1 79.3s0 79.be9754b3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e56) 2024-05-15T12:46:25.805 INFO:tasks.ceph.osd.3.smithi117.stderr:2024-05-15T12:46:25.720+0000 ea5d640 -1 osd.3 90 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4700.0:2 71.0 71:0384ab8e:::oid:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53) 2024-05-15T12:46:26.063 INFO:tasks.ceph.osd.7.smithi178.stderr:2024-05-15T12:46:25.995+0000 ea5d640 -1 osd.7 91 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4658.0:2 65.15 65.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e52) 2024-05-15T12:46:26.179 INFO:tasks.ceph.osd.1.smithi117.stderr:2024-05-15T12:46:26.185+0000 12a65640 -1 osd.1 89 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4706.0:1 79.3s0 79.be9754b3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e56) 2024-05-15T12:46:27.099 INFO:tasks.ceph.osd.7.smithi178.stderr:2024-05-15T12:46:27.080+0000 ea5d640 -1 osd.7 92 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4691.0:1 80.0s0 80.fc7118c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e56) 2024-05-15T12:46:27.406 INFO:tasks.ceph.osd.1.smithi117.stderr:2024-05-15T12:46:27.394+0000 12a65640 -1 osd.1 90 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4706.0:1 79.3s0 79.be9754b3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e56) 2024-05-15T12:46:28.141 INFO:tasks.ceph.osd.7.smithi178.stderr:2024-05-15T12:46:28.136+0000 ea5d640 -1 osd.7 93 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4658.0:2 65.15 65.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e52) 2024-05-15T12:46:28.552 INFO:tasks.ceph.osd.1.smithi117.stderr:2024-05-15T12:46:28.552+0000 12a65640 -1 osd.1 91 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4706.0:1 79.3s0 79.be9754b3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e56) 2024-05-15T12:46:29.130 INFO:tasks.ceph.osd.7.smithi178.stderr:2024-05-15T12:46:29.132+0000 ea5d640 -1 osd.7 93 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4658.0:2 65.15 65.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e52) 2024-05-15T12:46:29.636 INFO:tasks.ceph.osd.1.smithi117.stderr:2024-05-15T12:46:29.607+0000 12a65640 -1 osd.1 92 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4706.0:1 79.3s0 79.be9754b3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e56) 2024-05-15T12:46:30.753 INFO:tasks.ceph.osd.7.smithi178.stderr:2024-05-15T12:46:30.759+0000 ea5d640 -1 osd.7 95 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4658.0:2 65.15 65.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e52) 2024-05-15T12:46:30.799 INFO:tasks.ceph.osd.1.smithi117.stderr:2024-05-15T12:46:30.777+0000 12a65640 -1 osd.1 94 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4706.0:1 79.3s0 79.be9754b3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e56) 2024-05-15T12:46:31.946 INFO:tasks.ceph.osd.1.smithi117.stderr:2024-05-15T12:46:31.835+0000 12a65640 -1 osd.1 94 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4706.0:1 79.3s0 79.be9754b3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e56) 2024-05-15T12:46:32.069 INFO:tasks.ceph.osd.7.smithi178.stderr:2024-05-15T12:46:32.034+0000 ea5d640 -1 osd.7 96 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4658.0:2 65.15 65.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e52) 2024-05-15T12:46:32.996 INFO:tasks.ceph.osd.1.smithi117.stderr:2024-05-15T12:46:32.908+0000 12a65640 -1 osd.1 96 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4706.0:1 79.3s0 79:cd2ae97d:::foo1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) 2024-05-15T12:46:33.088 INFO:tasks.ceph.osd.7.smithi178.stderr:2024-05-15T12:46:33.090+0000 ea5d640 -1 osd.7 97 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4658.0:2 65.15 65.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e52) 2024-05-15T12:46:33.959 INFO:tasks.ceph.osd.1.smithi117.stderr:2024-05-15T12:46:33.960+0000 12a65640 -1 osd.1 96 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4706.0:1 79.3s0 79:cd2ae97d:::foo1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) 2024-05-15T12:46:34.209 INFO:tasks.ceph.osd.7.smithi178.stderr:2024-05-15T12:46:34.190+0000 ea5d640 -1 osd.7 97 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4658.0:2 65.15 65:ad551702:::test:head [create,setxattr key (5) in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52) 2024-05-15T12:46:35.030 INFO:tasks.ceph.osd.1.smithi117.stderr:2024-05-15T12:46:35.028+0000 12a65640 -1 osd.1 97 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4706.0:1 79.3s0 79:cd2ae97d:::foo1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) 2024-05-15T12:46:35.743 INFO:tasks.ceph.osd.7.smithi178.stderr:2024-05-15T12:46:35.699+0000 ea5d640 -1 osd.7 99 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4658.0:2 65.15 65:ad551702:::test:head [create,setxattr key (5) in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52) 2024-05-15T12:46:36.192 INFO:tasks.ceph.osd.1.smithi117.stderr:2024-05-15T12:46:36.157+0000 12a65640 -1 osd.1 98 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4706.0:1 79.3s0 79:cd2ae97d:::foo1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) 2024-05-15T12:46:37.244 INFO:tasks.ceph.osd.1.smithi117.stderr:2024-05-15T12:46:37.240+0000 12a65640 -1 osd.1 99 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4706.0:1 79.3s0 79:cd2ae97d:::foo1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) 2024-05-15T12:46:37.247 INFO:tasks.workunit.client.0.smithi117.stdout: watch_notify: Running main() from gmock_main.cc 2024-05-15T12:46:37.247 INFO:tasks.workunit.client.0.smithi117.stdout: watch_notify: [==========] Running 2 tests from 1 test suite. 2024-05-15T12:46:37.247 INFO:tasks.workunit.client.0.smithi117.stdout: watch_notify: [----------] Global test environment set-up. 2024-05-15T12:46:37.247 INFO:tasks.workunit.client.0.smithi117.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify 2024-05-15T12:46:37.247 INFO:tasks.workunit.client.0.smithi117.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotify 2024-05-15T12:46:37.247 INFO:tasks.workunit.client.0.smithi117.stdout: watch_notify: handle_notify cookie 94918320976336 notify_id 188978561024 notifier_gid 4546 2024-05-15T12:46:37.248 INFO:tasks.workunit.client.0.smithi117.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotify (48128 ms) 2024-05-15T12:46:37.248 INFO:tasks.workunit.client.0.smithi117.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotifyTimeout 2024-05-15T12:46:37.248 INFO:tasks.workunit.client.0.smithi117.stdout: watch_notify: Trying... 2024-05-15T12:46:37.248 INFO:tasks.workunit.client.0.smithi117.stdout: watch_notify: handle_notify cookie 94918333764288 notify_id 390842023937 notifier_gid 4735 2024-05-15T12:46:37.248 INFO:tasks.workunit.client.0.smithi117.stdout: watch_notify: Waiting for 3.000000000s 2024-05-15T12:46:37.248 INFO:tasks.workunit.client.0.smithi117.stdout: watch_notify: Timed out. 2024-05-15T12:46:37.248 INFO:tasks.workunit.client.0.smithi117.stdout: watch_notify: Flushing... 2024-05-15T12:46:37.248 INFO:tasks.workunit.client.0.smithi117.stdout: watch_notify: Flushed... 2024-05-15T12:46:37.248 INFO:tasks.workunit.client.0.smithi117.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotifyTimeout (63044 ms) 2024-05-15T12:46:37.248 INFO:tasks.workunit.client.0.smithi117.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify (111172 ms total) 2024-05-15T12:46:37.248 INFO:tasks.workunit.client.0.smithi117.stdout: watch_notify: 2024-05-15T12:46:37.248 INFO:tasks.workunit.client.0.smithi117.stdout: watch_notify: [----------] Global test environment tear-down 2024-05-15T12:46:37.248 INFO:tasks.workunit.client.0.smithi117.stdout: watch_notify: [==========] 2 tests from 1 test suite ran. (111172 ms total) 2024-05-15T12:46:37.248 INFO:tasks.workunit.client.0.smithi117.stdout: watch_notify: [ PASSED ] 2 tests. 2024-05-15T12:46:38.361 INFO:tasks.ceph.osd.1.smithi117.stderr:2024-05-15T12:46:38.364+0000 12a65640 -1 osd.1 100 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4706.0:1 79.3s0 79:cd2ae97d:::foo1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) 2024-05-15T12:46:57.444 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: Running main() from gmock_main.cc 2024-05-15T12:46:57.445 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [==========] Running 37 tests from 2 test suites. 2024-05-15T12:46:57.445 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [----------] Global test environment set-up. 2024-05-15T12:46:57.445 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP 2024-05-15T12:46:57.445 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: seed 39980 2024-05-15T12:46:57.445 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2024-05-15T12:46:57.445 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2024-05-15T12:46:57.445 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2024-05-15T12:46:57.445 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (31287 ms) 2024-05-15T12:46:57.445 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2024-05-15T12:46:57.445 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (9247 ms) 2024-05-15T12:46:57.445 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2024-05-15T12:46:57.445 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (191 ms) 2024-05-15T12:46:57.446 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2024-05-15T12:46:57.446 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (78 ms) 2024-05-15T12:46:57.446 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2024-05-15T12:46:57.446 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (40 ms) 2024-05-15T12:46:57.446 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2024-05-15T12:46:57.446 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (294 ms) 2024-05-15T12:46:57.446 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2024-05-15T12:46:57.446 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (480 ms) 2024-05-15T12:46:57.446 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2024-05-15T12:46:57.446 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (180 ms) 2024-05-15T12:46:57.446 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2024-05-15T12:46:57.446 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (113 ms) 2024-05-15T12:46:57.446 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2024-05-15T12:46:57.446 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (48 ms) 2024-05-15T12:46:57.447 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2024-05-15T12:46:57.447 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (101 ms) 2024-05-15T12:46:57.447 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2024-05-15T12:46:57.447 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (254 ms) 2024-05-15T12:46:57.447 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2024-05-15T12:46:57.447 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (99 ms) 2024-05-15T12:46:57.447 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2024-05-15T12:46:57.447 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (230 ms) 2024-05-15T12:46:57.447 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2024-05-15T12:46:57.447 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (506 ms) 2024-05-15T12:46:57.447 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-05-15T12:46:57.447 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (1560 ms) 2024-05-15T12:46:57.447 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2024-05-15T12:46:57.448 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (897 ms) 2024-05-15T12:46:57.448 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2024-05-15T12:46:57.448 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (458 ms) 2024-05-15T12:46:57.448 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2024-05-15T12:46:57.448 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (403 ms) 2024-05-15T12:46:57.448 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP (46466 ms total) 2024-05-15T12:46:57.448 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: 2024-05-15T12:46:57.448 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2024-05-15T12:46:57.448 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2024-05-15T12:46:57.448 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (58812 ms) 2024-05-15T12:46:57.448 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2024-05-15T12:46:57.448 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (11507 ms) 2024-05-15T12:46:57.448 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2024-05-15T12:46:57.448 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (2407 ms) 2024-05-15T12:46:57.449 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2024-05-15T12:46:57.449 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (363 ms) 2024-05-15T12:46:57.449 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2024-05-15T12:47:01.887 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP2 ( cls: Running main() from gmock_main.cc 2024-05-15T12:47:01.887 INFO:tasks.workunit.client.0.smithi117.stdout: cls: [==========] Running 2 tests from 1 test suite. 2024-05-15T12:47:01.887 INFO:tasks.workunit.client.0.smithi117.stdout: cls: [----------] Global test environment set-up. 2024-05-15T12:47:01.887 INFO:tasks.workunit.client.0.smithi117.stdout: cls: [----------] 2 tests from NeoRadosCls 2024-05-15T12:47:01.887 INFO:tasks.workunit.client.0.smithi117.stdout: cls: [ RUN ] NeoRadosCls.DNE 2024-05-15T12:47:01.887 INFO:tasks.workunit.client.0.smithi117.stdout: cls: [ OK ] NeoRadosCls.DNE (41250 ms) 2024-05-15T12:47:01.887 INFO:tasks.workunit.client.0.smithi117.stdout: cls: [ RUN ] NeoRadosCls.RemoteReads 2024-05-15T12:47:01.888 INFO:tasks.workunit.client.0.smithi117.stdout: cls: [ OK ] NeoRadosCls.RemoteReads (94693 ms) 2024-05-15T12:47:01.888 INFO:tasks.workunit.client.0.smithi117.stdout: cls: [----------] 2 tests from NeoRadosCls (135943 ms total) 2024-05-15T12:47:01.888 INFO:tasks.workunit.client.0.smithi117.stdout: cls: 2024-05-15T12:47:01.888 INFO:tasks.workunit.client.0.smithi117.stdout: cls: [----------] Global test environment tear-down 2024-05-15T12:47:01.888 INFO:tasks.workunit.client.0.smithi117.stdout: cls: [==========] 2 tests from 1 test suite ran. (135943 ms total) 2024-05-15T12:47:01.888 INFO:tasks.workunit.client.0.smithi117.stdout: cls: [ PASSED ] 2 tests. 2024-05-15T12:47:05.338 INFO:tasks.workunit.client.0.smithi117.stdout: cmd: Running main() from gmock_main.cc 2024-05-15T12:47:05.338 INFO:tasks.workunit.client.0.smithi117.stdout: cmd: [==========] Running 3 tests from 1 test suite. 2024-05-15T12:47:05.338 INFO:tasks.workunit.client.0.smithi117.stdout: cmd: [----------] Global test environment set-up. 2024-05-15T12:47:05.338 INFO:tasks.workunit.client.0.smithi117.stdout: cmd: [----------] 3 tests from NeoRadosCmd 2024-05-15T12:47:05.338 INFO:tasks.workunit.client.0.smithi117.stdout: cmd: [ RUN ] NeoRadosCmd.MonDescribe 2024-05-15T12:47:05.338 INFO:tasks.workunit.client.0.smithi117.stdout: cmd: [ OK ] NeoRadosCmd.MonDescribe (6112 ms) 2024-05-15T12:47:05.338 INFO:tasks.workunit.client.0.smithi117.stdout: cmd: [ RUN ] NeoRadosCmd.OSDCmd 2024-05-15T12:47:05.338 INFO:tasks.workunit.client.0.smithi117.stdout: cmd: [ OK ] NeoRadosCmd.OSDCmd (13646 ms) 2024-05-15T12:47:05.338 INFO:tasks.workunit.client.0.smithi117.stdout: cmd: [ RUN ] NeoRadosCmd.PGCmd 2024-05-15T12:47:05.339 INFO:tasks.workunit.client.0.smithi117.stdout: cmd: [ OK ] NeoRadosCmd.PGCmd (119637 ms) 2024-05-15T12:47:05.339 INFO:tasks.workunit.client.0.smithi117.stdout: cmd: [----------] 3 tests from NeoRadosCmd (139395 ms total) 2024-05-15T12:47:05.339 INFO:tasks.workunit.client.0.smithi117.stdout: cmd: 2024-05-15T12:47:05.339 INFO:tasks.workunit.client.0.smithi117.stdout: cmd: [----------] Global test environment tear-down 2024-05-15T12:47:05.339 INFO:tasks.workunit.client.0.smithi117.stdout: cmd: [==========] 3 tests from 1 test suite ran. (139395 ms total) 2024-05-15T12:47:05.339 INFO:tasks.workunit.client.0.smithi117.stdout: cmd: [ PASSED ] 3 tests. 2024-05-15T12:47:07.836 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2024-05-15T12:47:07.836 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2024-05-15T12:47:07.836 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2024-05-15T12:47:07.836 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2024-05-15T12:47:07.836 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2024-05-15T12:47:07.836 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: notify 2024-05-15T12:47:07.836 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (36043 ms) 2024-05-15T12:47:07.836 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2024-05-15T12:47:07.837 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (900 ms) 2024-05-15T12:47:07.837 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (36943 ms total) 2024-05-15T12:47:07.837 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: 2024-05-15T12:47:07.837 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2024-05-15T12:47:07.837 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2024-05-15T12:47:07.837 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: notify 2024-05-15T12:47:07.837 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (38667 ms) 2024-05-15T12:47:07.837 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2024-05-15T12:47:07.837 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: notify 2024-05-15T12:47:07.837 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (30842 ms) 2024-05-15T12:47:07.837 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2024-05-15T12:47:07.837 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (1713 ms) 2024-05-15T12:47:07.837 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2024-05-15T12:47:07.837 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (3991 ms) 2024-05-15T12:47:07.837 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2024-05-15T12:47:07.838 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: handle_notify cookie 94789781450640 notify_id 476741369858 notifier_gid 4807 2024-05-15T12:47:07.838 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (1358 ms) 2024-05-15T12:47:07.838 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2024-05-15T12:47:07.838 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: handle_notify cookie 94789781450640 notify_id 476741369858 notifier_gid 4807 2024-05-15T12:47:07.838 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (1043 ms) 2024-05-15T12:47:07.838 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2024-05-15T12:47:07.838 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: handle_notify cookie 94789781453856 notify_id 481036337152 notifier_gid 4807 2024-05-15T12:47:07.838 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (3773 ms) 2024-05-15T12:47:07.838 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2024-05-15T12:47:07.838 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: handle_notify cookie 94789781453856 notify_id 481036337154 notifier_gid 4807 2024-05-15T12:47:07.838 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (2500 ms) 2024-05-15T12:47:07.838 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2024-05-15T12:47:07.838 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: handle_notify cookie 94789781453856 notify_id 485331304451 notifier_gid 4807 2024-05-15T12:47:07.838 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: handle_notify cookie 94789781453856 notify_id 485331304452 notifier_gid 4807 2024-05-15T12:47:07.839 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (444 ms) 2024-05-15T12:47:07.839 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2024-05-15T12:47:07.839 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: handle_notify cookie 94789780204912 notify_id 485331304452 notifier_gid 4807 2024-05-15T12:47:07.839 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (296 ms) 2024-05-15T12:47:07.839 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2024-05-15T12:47:07.839 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: trying... 2024-05-15T12:47:09.292 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: handle_notify cookie 94789780204912 noti api_c_read_operations: Running main() from gmock_main.cc 2024-05-15T12:47:09.292 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2024-05-15T12:47:09.292 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [----------] Global test environment set-up. 2024-05-15T12:47:09.292 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2024-05-15T12:47:09.293 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2024-05-15T12:47:09.293 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2024-05-15T12:47:09.293 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2024-05-15T12:47:09.293 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (36513 ms) 2024-05-15T12:47:09.293 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2024-05-15T12:47:09.293 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (1034 ms) 2024-05-15T12:47:09.294 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2024-05-15T12:47:09.294 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (789 ms) 2024-05-15T12:47:09.294 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2024-05-15T12:47:09.294 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (539 ms) 2024-05-15T12:47:09.294 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2024-05-15T12:47:09.295 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (102 ms) 2024-05-15T12:47:09.295 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2024-05-15T12:47:09.295 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (271 ms) 2024-05-15T12:47:09.295 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2024-05-15T12:47:09.295 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (125 ms) 2024-05-15T12:47:09.295 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2024-05-15T12:47:09.296 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (227 ms) 2024-05-15T12:47:09.296 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2024-05-15T12:47:09.296 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (1660 ms) 2024-05-15T12:47:09.296 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2024-05-15T12:47:09.296 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (1369 ms) 2024-05-15T12:47:09.296 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2024-05-15T12:47:09.296 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (876 ms) 2024-05-15T12:47:09.296 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2024-05-15T12:47:09.297 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (100 ms) 2024-05-15T12:47:09.297 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2024-05-15T12:47:09.297 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (84247 ms) 2024-05-15T12:47:09.297 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2024-05-15T12:47:09.297 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (589 ms) 2024-05-15T12:47:09.297 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2024-05-15T12:47:09.297 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (295 ms) 2024-05-15T12:47:09.297 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2024-05-15T12:47:09.298 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (83 ms) 2024-05-15T12:47:09.298 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (128819 ms total) 2024-05-15T12:47:09.298 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: 2024-05-15T12:47:09.298 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [----------] Global test environment tear-down 2024-05-15T12:47:09.298 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (143433 ms total) 2024-05-15T12:47:09.298 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2024-05-15T12:47:15.019 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:05:22.198 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:15.035 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:05:22.212 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:15.698 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:05:22.876 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:16.067 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:05:23.239 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:16.092 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:05:23.264 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:16.097 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:05:23.276 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:16.112 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:05:23.290 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:16.532 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:05:23.703 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:16.578 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:05:23.756 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:18.798 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [==========] Running 11 tests from 3 test suites. 2024-05-15T12:47:18.798 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [----------] Global test environment set-up. 2024-05-15T12:47:18.798 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [----------] 7 tests from LibRadosList 2024-05-15T12:47:18.798 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2024-05-15T12:47:18.798 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [ OK ] LibRadosList.ListObjects (42146 ms) 2024-05-15T12:47:18.798 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2024-05-15T12:47:18.798 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (4068 ms) 2024-05-15T12:47:18.798 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2024-05-15T12:47:18.798 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: myset foo1,foo2,foo3 2024-05-15T12:47:18.799 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: foo1 2024-05-15T12:47:18.799 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: foo2 2024-05-15T12:47:18.799 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: foo3 2024-05-15T12:47:18.799 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: myset foo1,foo4,foo5 2024-05-15T12:47:18.799 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: foo4 2024-05-15T12:47:18.799 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: foo5 2024-05-15T12:47:18.799 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: foo1 2024-05-15T12:47:18.799 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: myset foo6,foo7 2024-05-15T12:47:18.799 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: foo7 2024-05-15T12:47:18.799 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: foo6 2024-05-15T12:47:18.799 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-05-15T12:47:18.799 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: ns1:foo4 2024-05-15T12:47:18.799 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: ns1:foo5 2024-05-15T12:47:18.799 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: ns2:foo7 2024-05-15T12:47:18.800 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: ns2:foo6 2024-05-15T12:47:18.800 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: ns1:foo1 2024-05-15T12:47:18.800 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: :foo1 2024-05-15T12:47:18.800 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: :foo2 2024-05-15T12:47:18.800 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: :foo3 2024-05-15T12:47:18.800 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (91893 ms) 2024-05-15T12:47:18.800 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2024-05-15T12:47:18.800 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 1 0 2024-05-15T12:47:18.800 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 10 0 2024-05-15T12:47:18.800 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 13 0 2024-05-15T12:47:18.800 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 7 0 2024-05-15T12:47:18.800 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 14 0 2024-05-15T12:47:18.800 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 0 0 2024-05-15T12:47:18.800 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 15 0 2024-05-15T12:47:18.800 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 11 0 2024-05-15T12:47:18.800 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 5 0 2024-05-15T12:47:18.801 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 8 0 2024-05-15T12:47:18.801 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 6 0 2024-05-15T12:47:18.801 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 3 0 2024-05-15T12:47:18.801 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 4 0 2024-05-15T12:47:18.801 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 12 0 2024-05-15T12:47:18.801 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 9 0 2024-05-15T12:47:18.801 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 2 0 2024-05-15T12:47:18.801 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-05-15T12:47:18.801 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (6911 ms) 2024-05-15T12:47:18.801 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2024-05-15T12:47:18.801 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: x cursor=MIN 2024-05-15T12:47:18.801 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > oid=1 cursor=5:02547ec2:::1:head 2024-05-15T12:47:18.801 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > oid=10 cursor=5:52ea6a34:::10:head 2024-05-15T12:47:18.801 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > oid=13 cursor=5:566253c9:::13:head 2024-05-15T12:47:18.801 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > oid=7 cursor=5:5c6b0b28:::7:head 2024-05-15T12:47:18.802 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > oid=14 cursor=5:62a1935d:::14:head 2024-05-15T12:47:18.802 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > oid=0 cursor=5:6cac518f:::0:head 2024-05-15T12:47:18.802 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > oid=15 cursor=5:863748b0:::15:head 2024-05-15T12:47:18.802 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > oid=11 cursor=5:89d3ae78:::11:head 2024-05-15T12:47:18.802 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > oid=5 cursor=5:b29083e3:::5:head 2024-05-15T12:47:18.802 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > oid=8 cursor=5:bd63b0f1:::8:head 2024-05-15T12:47:18.802 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > oid=6 cursor=5:c4fdafeb:::6:head 2024-05-15T12:47:18.802 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > oid=3 cursor=5:cfc208b3:::3:head 2024-05-15T12:47:18.802 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > oid=4 cursor=5:d83876eb:::4:head 2024-05-15T12:47:18.802 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > oid=12 cursor=5:de5d7c5f:::12:head 2024-05-15T12:47:18.802 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > oid=9 cursor=5:e960b815:::9:head 2024-05-15T12:47:18.802 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > oid=2 cursor=5:f905c69b:::2:head 2024-05-15T12:47:18.802 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: FIRST> seek to MIN oid=1 2024-05-15T12:47:18.802 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : oid=1 cursor=5:02547ec2:::1:head 2024-05-15T12:47:18.803 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 5:02547ec2:::1:head 2024-05-15T12:47:18.803 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 5:02547ec2:::1:head -> 1 2024-05-15T12:47:18.803 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : oid=10 cursor=5:52ea6a34:::10:head 2024-05-15T12:47:18.803 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 5:52ea6a34:::10:head 2024-05-15T12:47:18.803 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 5:52ea6a34:::10:head -> 10 2024-05-15T12:47:18.803 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : oid=13 cursor=5:566253c9:::13:head 2024-05-15T12:47:18.803 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 5:566253c9:::13:head 2024-05-15T12:47:18.803 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 5:566253c9:::13:head -> 13 2024-05-15T12:47:18.803 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : oid=7 cursor=5:5c6b0b28:::7:head 2024-05-15T12:47:18.803 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 5:5c6b0b28:::7:head 2024-05-15T12:47:18.803 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 5:5c6b0b28:::7:head -> 7 2024-05-15T12:47:21.286 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : oid=14 cursor=5:62a1935d:::14:head 2024-05-15T12:47:21.286 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 5:62a1935d:::14:head 2024-05-15T12:47:21.286 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 5:62a1935d:::14:head -> 14 2024-05-15T12:47:21.287 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : oid=0 cursor=5:6cac518f:::0:head 2024-05-15T12:47:21.287 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 5:6cac518f:::0:head 2024-05-15T12:47:21.287 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 5:6cac518f:::0:head -> 0 2024-05-15T12:47:21.287 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : oid=15 cursor=5:863748b0:::15:head 2024-05-15T12:47:21.287 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 5:863748b0:::15:head 2024-05-15T12:47:21.287 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 5:863748b0:::15:head -> 15 2024-05-15T12:47:21.287 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : oid=11 cursor=5:89d3ae78:::11:head 2024-05-15T12:47:21.287 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 5:89d3ae78:::11:head 2024-05-15T12:47:21.287 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 5:89d3ae78:::11:head -> 11 2024-05-15T12:47:21.287 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : oid=5 cursor=5:b29083e3:::5:head 2024-05-15T12:47:21.287 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 5:b29083e3:::5:head 2024-05-15T12:47:21.287 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 5:b29083e3:::5:head -> 5 2024-05-15T12:47:21.287 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : oid=8 cursor=5:bd63b0f1:::8:head 2024-05-15T12:47:21.287 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 5:bd63b0f1:::8:head 2024-05-15T12:47:21.287 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 5:bd63b0f1:::8:head -> 8 2024-05-15T12:47:21.288 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : oid=6 cursor=5:c4fdafeb:::6:head 2024-05-15T12:47:21.288 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 5:c4fdafeb:::6:head 2024-05-15T12:47:21.288 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 5:c4fdafeb:::6:head -> 6 2024-05-15T12:47:21.288 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : oid=3 cursor=5:cfc208b3:::3:head 2024-05-15T12:47:21.288 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 5:cfc208b3:::3:head 2024-05-15T12:47:21.288 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 5:cfc208b3:::3:head -> 3 2024-05-15T12:47:21.288 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : oid=4 cursor=5:d83876eb:::4:head 2024-05-15T12:47:21.288 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 5:d83876eb:::4:head 2024-05-15T12:47:21.288 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 5:d83876eb:::4:head -> 4 2024-05-15T12:47:21.288 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : oid=12 cursor=5:de5d7c5f:::12:head 2024-05-15T12:47:21.288 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 5:de5d7c5f:::12:head 2024-05-15T12:47:21.288 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 5:de5d7c5f:::12:head -> 12 2024-05-15T12:47:21.288 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : oid=9 cursor=5:e960b815:::9:head 2024-05-15T12:47:21.288 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 5:e960b815:::9:head 2024-05-15T12:47:21.288 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 5:e960b815:::9:head -> 9 2024-05-15T12:47:21.288 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : oid=2 cursor=5:f905c69b:::2:head 2024-05-15T12:47:21.289 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 5:f905c69b:::2:head 2024-05-15T12:47:21.289 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 5:f905c69b:::2:head -> 2 2024-05-15T12:47:21.289 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 5:52ea6a34:::10:head 2024-05-15T12:47:21.289 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : cursor()=5:52ea6a34:::10:head expected=5:52ea6a34:::10:head 2024-05-15T12:47:21.289 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 5:52ea6a34:::10:head -> 10 2024-05-15T12:47:21.289 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : entry=10 expected=10 2024-05-15T12:47:21.289 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 5:566253c9:::13:head 2024-05-15T12:47:21.289 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : cursor()=5:566253c9:::13:head expected=5:566253c9:::13:head 2024-05-15T12:47:21.289 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 5:566253c9:::13:head -> 13 2024-05-15T12:47:21.289 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : entry=13 expected=13 2024-05-15T12:47:21.289 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 5:02547ec2:::1:head 2024-05-15T12:47:21.289 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : cursor()=5:02547ec2:::1:head expected=5:02547ec2:::1:head 2024-05-15T12:47:21.289 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 5:02547ec2:::1:head -> 1 2024-05-15T12:47:21.289 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : entry=1 expected=1 2024-05-15T12:47:21.289 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 5:62a1935d:::14:head 2024-05-15T12:47:21.289 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : cursor()=5:62a1935d:::14:head expected=5:62a1935d:::14:head 2024-05-15T12:47:21.290 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 5:62a1935d:::14:head -> 14 2024-05-15T12:47:21.290 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : entry=14 expected=14 2024-05-15T12:47:21.290 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 5:5c6b0b28:::7:head 2024-05-15T12:47:21.290 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : cursor()=5:5c6b0b28:::7:head expected=5:5c6b0b28:::7:head 2024-05-15T12:47:21.290 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 5:5c6b0b28:::7:head -> 7 2024-05-15T12:47:21.290 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : entry=7 expected=7 2024-05-15T12:47:21.290 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 5:e960b815:::9:head 2024-05-15T12:47:21.290 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : cursor()=5:e960b815:::9:head expected=5:e960b815:::9:head 2024-05-15T12:47:21.290 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 5:e960b815:::9:head -> 9 2024-05-15T12:47:21.290 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : entry=9 expected=9 2024-05-15T12:47:21.290 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 5:de5d7c5f:::12:head 2024-05-15T12:47:21.290 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : cursor()=5:de5d7c5f:::12:head expected=5:de5d7c5f:::12:head 2024-05-15T12:47:21.290 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 5:de5d7c5f:::12:head -> 12 2024-05-15T12:47:21.290 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : entry=12 expected=12 2024-05-15T12:47:21.291 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 5:d83876eb:::4:head 2024-05-15T12:47:21.291 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : cursor()=5:d83876eb:::4:head expected=5:d83876eb:::4:head 2024-05-15T12:47:21.291 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 5:d83876eb:::4:head -> 4 2024-05-15T12:47:21.291 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : entry=4 expected=4 2024-05-15T12:47:21.291 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 5:863748b0:::15:head 2024-05-15T12:47:24.512 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: process_1_[40786]: starting. 2024-05-15T12:47:24.512 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: process_1_[40786]: creating pool ceph_test_rados_delete_pools_parallel.smithi117-40725 2024-05-15T12:47:24.513 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: process_1_[40786]: created object 0... 2024-05-15T12:47:24.513 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: process_1_[40786]: created object 25... 2024-05-15T12:47:24.513 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: process_1_[40786]: created object 49... 2024-05-15T12:47:24.513 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: process_1_[40786]: finishing. 2024-05-15T12:47:24.513 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: process_1_[40786]: shutting down. 2024-05-15T12:47:24.513 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: process_2_[40795]: starting. 2024-05-15T12:47:24.513 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: process_2_[40795]: deleting pool ceph_test_rados_delete_pools_parallel.smithi117-40725 2024-05-15T12:47:24.513 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: process_2_[40795]: shutting down. 2024-05-15T12:47:24.513 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: ******************************* 2024-05-15T12:47:24.513 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: process_3_[42377]: starting. 2024-05-15T12:47:24.513 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: process_3_[42377]: creating pool ceph_test_rados_delete_pools_parallel.smithi117-40725 2024-05-15T12:47:24.513 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: process_3_[42377]: created object 0... 2024-05-15T12:47:24.513 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: process_3_[42377]: created object 25... 2024-05-15T12:47:24.513 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: process_3_[42377]: created object 49... 2024-05-15T12:47:24.513 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: process_3_[42377]: finishing. 2024-05-15T12:47:24.514 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: process_3_[42377]: shutting down. 2024-05-15T12:47:24.514 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: ******************************* 2024-05-15T12:47:24.514 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: process_5_[42379]: starting. 2024-05-15T12:47:24.514 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: process_5_[42379]: listing objects. 2024-05-15T12:47:24.514 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: process_5_[42379]: listed object 0... 2024-05-15T12:47:24.514 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: process_5_[42379]: listed object 25... 2024-05-15T12:47:24.514 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: process_5_[42379]: saw 50 objects 2024-05-15T12:47:24.514 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: process_5_[42379]: shutting down. 2024-05-15T12:47:24.514 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: ******************************* 2024-05-15T12:47:24.514 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: process_4_[42378]: starting. 2024-05-15T12:47:24.514 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: process_4_[42378]: deleting pool ceph_test_rados_delete_pools_parallel.smithi117-40725 2024-05-15T12:47:24.514 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: process_4_[42378]: shutting down. 2024-05-15T12:47:24.514 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: ******************************* 2024-05-15T12:47:24.514 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: ******************************* 2024-05-15T12:47:24.515 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: ******* SUCCESS ********** 2024-05-15T12:47:26.239 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: Running main() from gmock_main.cc 2024-05-15T12:47:26.239 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2024-05-15T12:47:26.239 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: [----------] Global test environment set-up. 2024-05-15T12:47:26.239 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: [----------] 5 tests from LibRadosStat 2024-05-15T12:47:26.239 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2024-05-15T12:47:26.239 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: [ OK ] LibRadosStat.Stat (35932 ms) 2024-05-15T12:47:26.239 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2024-05-15T12:47:26.239 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (2200 ms) 2024-05-15T12:47:26.239 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2024-05-15T12:47:26.239 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: [ OK ] LibRadosStat.StatNS (1271 ms) 2024-05-15T12:47:26.240 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2024-05-15T12:47:26.240 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (3 ms) 2024-05-15T12:47:26.240 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2024-05-15T12:47:26.240 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (3509 ms) 2024-05-15T12:47:26.240 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: [----------] 5 tests from LibRadosStat (42916 ms total) 2024-05-15T12:47:26.240 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: 2024-05-15T12:47:26.240 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2024-05-15T12:47:26.240 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2024-05-15T12:47:26.240 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (48253 ms) 2024-05-15T12:47:26.240 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2024-05-15T12:47:26.240 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (16097 ms) 2024-05-15T12:47:26.241 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2024-05-15T12:47:26.241 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (1041 ms) 2024-05-15T12:47:26.241 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2024-05-15T12:47:26.241 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (3260 ms) 2024-05-15T12:47:26.241 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (68651 ms total) 2024-05-15T12:47:26.241 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: 2024-05-15T12:47:26.241 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: [----------] Global test environment tear-down 2024-05-15T12:47:26.241 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (160551 ms total) 2024-05-15T12:47:26.241 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: [ PASSED ] 9 tests. 2024-05-15T12:47:26.283 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: Running main() from gmock_main.cc 2024-05-15T12:47:26.283 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2024-05-15T12:47:26.283 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: [----------] Global test environment set-up. 2024-05-15T12:47:26.283 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2024-05-15T12:47:26.283 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: seed 40420 2024-05-15T12:47:26.283 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2024-05-15T12:47:26.283 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (33703 ms) 2024-05-15T12:47:26.283 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2024-05-15T12:47:26.284 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (4153 ms) 2024-05-15T12:47:26.284 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2024-05-15T12:47:26.284 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (1659 ms) 2024-05-15T12:47:26.284 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2024-05-15T12:47:26.284 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (3572 ms) 2024-05-15T12:47:26.284 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2024-05-15T12:47:26.284 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (1494 ms) 2024-05-15T12:47:26.284 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (44581 ms total) 2024-05-15T12:47:26.284 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: 2024-05-15T12:47:26.284 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2024-05-15T12:47:26.284 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2024-05-15T12:47:26.284 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (54478 ms) 2024-05-15T12:47:26.285 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2024-05-15T12:47:26.285 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (101 ms) 2024-05-15T12:47:26.285 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2024-05-15T12:47:26.285 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (1707 ms) 2024-05-15T12:47:26.285 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2024-05-15T12:47:26.285 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (15855 ms) 2024-05-15T12:47:26.285 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (72141 ms total) 2024-05-15T12:47:26.285 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: 2024-05-15T12:47:26.285 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: [----------] Global test environment tear-down 2024-05-15T12:47:26.285 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (160506 ms total) 2024-05-15T12:47:26.285 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: [ PASSED ] 9 tests. 2024-05-15T12:47:29.830 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: Running main() from gmock_main.cc 2024-05-15T12:47:29.831 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2024-05-15T12:47:29.831 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [----------] Global test environment set-up. 2024-05-15T12:47:29.831 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2024-05-15T12:47:29.831 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: seed 40102 2024-05-15T12:47:29.831 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2024-05-15T12:47:29.831 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (17475 ms) 2024-05-15T12:47:29.831 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2024-05-15T12:47:29.831 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (19774 ms) 2024-05-15T12:47:29.831 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2024-05-15T12:47:29.831 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (1223 ms) 2024-05-15T12:47:29.831 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2024-05-15T12:47:29.831 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (1155 ms) 2024-05-15T12:47:29.831 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2024-05-15T12:47:29.832 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (2177 ms) 2024-05-15T12:47:29.832 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2024-05-15T12:47:29.832 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (1101 ms) 2024-05-15T12:47:29.832 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2024-05-15T12:47:29.832 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (250 ms) 2024-05-15T12:47:29.832 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2024-05-15T12:47:29.832 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (96 ms) 2024-05-15T12:47:29.832 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (43251 ms total) 2024-05-15T12:47:29.832 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: 2024-05-15T12:47:29.832 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2024-05-15T12:47:29.832 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2024-05-15T12:47:29.832 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (49138 ms) 2024-05-15T12:47:29.832 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2024-05-15T12:47:29.832 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (16184 ms) 2024-05-15T12:47:29.832 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2024-05-15T12:47:29.833 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (3860 ms) 2024-05-15T12:47:29.833 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2024-05-15T12:47:29.833 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (5654 ms) 2024-05-15T12:47:29.833 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2024-05-15T12:47:29.833 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (184 ms) 2024-05-15T12:47:29.833 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2024-05-15T12:47:29.833 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (362 ms) 2024-05-15T12:47:29.833 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2024-05-15T12:47:29.833 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (273 ms) 2024-05-15T12:47:29.833 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2024-05-15T12:47:29.833 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (309 ms) 2024-05-15T12:47:29.833 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (75964 ms total) 2024-05-15T12:47:29.833 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: 2024-05-15T12:47:29.833 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [----------] Global test environment tear-down 2024-05-15T12:47:29.833 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (164193 ms total) 2024-05-15T12:47:29.833 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ PASSED ] 16 tests. 2024-05-15T12:47:31.294 INFO:tasks.workunit.client.0.smithi117.stdout: open_pools_parallel: process_1_[40830]: starting. 2024-05-15T12:47:31.294 INFO:tasks.workunit.client.0.smithi117.stdout: open_pools_parallel: process_1_[40830]: creating pool ceph_test_rados_open_pools_parallel.smithi117-40747 2024-05-15T12:47:31.294 INFO:tasks.workunit.client.0.smithi117.stdout: open_pools_parallel: process_1_[40830]: created object 0... 2024-05-15T12:47:31.294 INFO:tasks.workunit.client.0.smithi117.stdout: open_pools_parallel: process_1_[40830]: created object 25... 2024-05-15T12:47:31.294 INFO:tasks.workunit.client.0.smithi117.stdout: open_pools_parallel: process_1_[40830]: created object 49... 2024-05-15T12:47:31.294 INFO:tasks.workunit.client.0.smithi117.stdout: open_pools_parallel: process_1_[40830]: finishing. 2024-05-15T12:47:31.294 INFO:tasks.workunit.client.0.smithi117.stdout: open_pools_parallel: process_1_[40830]: shutting down. 2024-05-15T12:47:31.294 INFO:tasks.workunit.client.0.smithi117.stdout: open_pools_parallel: process_2_[40833]: starting. 2024-05-15T12:47:31.294 INFO:tasks.workunit.client.0.smithi117.stdout: open_pools_parallel: process_2_[40833]: rados_pool_create. 2024-05-15T12:47:31.295 INFO:tasks.workunit.client.0.smithi117.stdout: open_pools_parallel: process_2_[40833]: rados_ioctx_create. 2024-05-15T12:47:31.295 INFO:tasks.workunit.client.0.smithi117.stdout: open_pools_parallel: process_2_[40833]: shutting down. 2024-05-15T12:47:31.295 INFO:tasks.workunit.client.0.smithi117.stdout: open_pools_parallel: ******************************* 2024-05-15T12:47:31.295 INFO:tasks.workunit.client.0.smithi117.stdout: open_pools_parallel: process_3_[51803]: starting. 2024-05-15T12:47:31.295 INFO:tasks.workunit.client.0.smithi117.stdout: open_pools_parallel: process_3_[51803]: creating pool ceph_test_rados_open_pools_parallel.smithi117-40747 2024-05-15T12:47:31.295 INFO:tasks.workunit.client.0.smithi117.stdout: open_pools_parallel: process_3_[51803]: created object 0... 2024-05-15T12:47:31.295 INFO:tasks.workunit.client.0.smithi117.stdout: open_pools_parallel: process_3_[51803]: created object 25... 2024-05-15T12:47:31.295 INFO:tasks.workunit.client.0.smithi117.stdout: open_pools_parallel: process_3_[51803]: created object 49... 2024-05-15T12:47:31.295 INFO:tasks.workunit.client.0.smithi117.stdout: open_pools_parallel: process_3_[51803]: finishing. 2024-05-15T12:47:31.295 INFO:tasks.workunit.client.0.smithi117.stdout: open_pools_parallel: process_3_[51803]: shutting down. 2024-05-15T12:47:31.295 INFO:tasks.workunit.client.0.smithi117.stdout: open_pools_parallel: ******************************* 2024-05-15T12:47:31.295 INFO:tasks.workunit.client.0.smithi117.stdout: open_pools_parallel: process_4_[51804]: starting. 2024-05-15T12:47:31.295 INFO:tasks.workunit.client.0.smithi117.stdout: open_pools_parallel: process_4_[51804]: rados_pool_create. 2024-05-15T12:47:31.295 INFO:tasks.workunit.client.0.smithi117.stdout: open_pools_parallel: process_4_[51804]: rados_ioctx_create. 2024-05-15T12:47:31.295 INFO:tasks.workunit.client.0.smithi117.stdout: open_pools_parallel: process_4_[51804]: shutting down. 2024-05-15T12:47:31.296 INFO:tasks.workunit.client.0.smithi117.stdout: open_pools_parallel: ******************************* 2024-05-15T12:47:31.296 INFO:tasks.workunit.client.0.smithi117.stdout: open_pools_parallel: ******************************* 2024-05-15T12:47:31.296 INFO:tasks.workunit.client.0.smithi117.stdout: open_pools_parallel: ******* SUCCESS ********** 2024-05-15T12:47:33.440 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_1_[40750]: starting. 2024-05-15T12:47:33.441 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_1_[40750]: creating pool ceph_test_rados_list_parallel.smithi117-40669 2024-05-15T12:47:33.441 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_1_[40750]: created object 0... 2024-05-15T12:47:33.441 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_1_[40750]: created object 25... 2024-05-15T12:47:33.441 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_1_[40750]: created object 49... 2024-05-15T12:47:33.441 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_1_[40750]: finishing. 2024-05-15T12:47:33.441 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_1_[40750]: shutting down. 2024-05-15T12:47:33.441 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_2_[40752]: starting. 2024-05-15T12:47:33.441 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_2_[40752]: listing objects. 2024-05-15T12:47:33.441 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_2_[40752]: listed object 0... 2024-05-15T12:47:33.441 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_2_[40752]: listed object 25... 2024-05-15T12:47:33.441 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_2_[40752]: saw 50 objects 2024-05-15T12:47:33.442 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_2_[40752]: shutting down. 2024-05-15T12:47:33.442 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-05-15T12:47:33.442 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_3_[42633]: starting. 2024-05-15T12:47:33.442 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_3_[42633]: creating pool ceph_test_rados_list_parallel.smithi117-40669 2024-05-15T12:47:33.442 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_3_[42633]: created object 0... 2024-05-15T12:47:33.442 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_3_[42633]: created object 25... 2024-05-15T12:47:33.442 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_3_[42633]: created object 49... 2024-05-15T12:47:33.442 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_3_[42633]: finishing. 2024-05-15T12:47:33.442 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_3_[42633]: shutting down. 2024-05-15T12:47:33.442 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-05-15T12:47:33.442 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_4_[42634]: starting. 2024-05-15T12:47:33.442 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_4_[42634]: listing objects. 2024-05-15T12:47:33.442 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_4_[42634]: listed object 0... 2024-05-15T12:47:33.443 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_4_[42634]: listed object 25... 2024-05-15T12:47:33.443 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_4_[42634]: saw 44 objects 2024-05-15T12:47:33.443 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_4_[42634]: shutting down. 2024-05-15T12:47:33.443 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-05-15T12:47:33.443 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_5_[42635]: starting. 2024-05-15T12:47:33.443 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_5_[42635]: removed 25 objects... 2024-05-15T12:47:33.443 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_5_[42635]: removed half of the objects 2024-05-15T12:47:33.443 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_5_[42635]: removed 50 objects... 2024-05-15T12:47:33.443 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_5_[42635]: removed 50 objects 2024-05-15T12:47:33.443 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_5_[42635]: shutting down. 2024-05-15T12:47:33.443 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-05-15T12:47:33.443 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-05-15T12:47:33.443 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_6_[67193]: starting. 2024-05-15T12:47:33.443 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_6_[67193]: creating pool ceph_test_rados_list_parallel.smithi117-40669 2024-05-15T12:47:33.444 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_6_[67193]: created object 0... 2024-05-15T12:47:33.444 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_6_[67193]: created object 25... 2024-05-15T12:47:33.444 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_6_[67193]: created object 49... 2024-05-15T12:47:33.444 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_6_[67193]: finishing. 2024-05-15T12:47:33.444 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_6_[67193]: shutting down. 2024-05-15T12:47:33.444 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-05-15T12:47:33.444 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-05-15T12:47:33.444 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_7_[67194]: starting. 2024-05-15T12:47:33.444 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_7_[67194]: listing objects. 2024-05-15T12:47:33.444 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_7_[67194]: listed object 0... 2024-05-15T12:47:33.444 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_7_[67194]: listed object 25... 2024-05-15T12:47:33.444 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_7_[67194]: listed object 50... 2024-05-15T12:47:33.444 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_7_[67194]: saw 54 objects 2024-05-15T12:47:33.444 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_7_[67194]: shutting down. 2024-05-15T12:47:33.444 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-05-15T12:47:33.445 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-05-15T12:47:33.445 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_8_[67195]: starting. 2024-05-15T12:47:33.445 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_8_[67195]: added 25 objects... 2024-05-15T12:47:33.445 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_8_[67195]: added half of the objects 2024-05-15T12:47:33.445 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_8_[67195]: added 50 objects... 2024-05-15T12:47:33.445 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_8_[67195]: added 50 objects 2024-05-15T12:47:33.445 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_8_[67195]: shutting down. 2024-05-15T12:47:33.445 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-05-15T12:47:33.445 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-05-15T12:47:36.191 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ********************** api_io: Running main() from gmock_main.cc 2024-05-15T12:47:36.192 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2024-05-15T12:47:36.192 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [----------] Global test environment set-up. 2024-05-15T12:47:36.192 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [----------] 14 tests from LibRadosIo 2024-05-15T12:47:36.192 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2024-05-15T12:47:36.192 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (34902 ms) 2024-05-15T12:47:36.192 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2024-05-15T12:47:36.192 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2024-05-15T12:47:36.192 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2024-05-15T12:47:36.192 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: no timeout :/ 2024-05-15T12:47:36.192 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: no timeout :/ 2024-05-15T12:47:36.192 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: no timeout :/ 2024-05-15T12:47:36.192 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: no timeout :/ 2024-05-15T12:47:36.192 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: no timeout :/ 2024-05-15T12:47:36.192 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (3879 ms) 2024-05-15T12:47:36.192 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2024-05-15T12:47:36.192 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (1740 ms) 2024-05-15T12:47:36.193 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2024-05-15T12:47:36.193 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ OK ] LibRadosIo.Checksum (372 ms) 2024-05-15T12:47:36.193 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2024-05-15T12:47:36.193 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (314 ms) 2024-05-15T12:47:36.193 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2024-05-15T12:47:36.193 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (80 ms) 2024-05-15T12:47:36.193 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2024-05-15T12:47:36.193 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (296 ms) 2024-05-15T12:47:36.193 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2024-05-15T12:47:36.193 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (54 ms) 2024-05-15T12:47:36.193 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2024-05-15T12:47:36.193 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ OK ] LibRadosIo.TruncTest (245 ms) 2024-05-15T12:47:36.193 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2024-05-15T12:47:36.193 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (209 ms) 2024-05-15T12:47:36.193 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2024-05-15T12:47:36.194 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (149 ms) 2024-05-15T12:47:36.194 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2024-05-15T12:47:36.194 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ OK ] LibRadosIo.RmXattr (457 ms) 2024-05-15T12:47:36.194 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2024-05-15T12:47:36.194 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ OK ] LibRadosIo.XattrIter (2892 ms) 2024-05-15T12:47:36.194 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [----------] 14 tests from LibRadosIo (45591 ms total) 2024-05-15T12:47:36.194 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: 2024-05-15T12:47:36.194 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2024-05-15T12:47:36.194 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2024-05-15T12:47:36.194 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (54325 ms) 2024-05-15T12:47:36.194 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2024-05-15T12:47:36.194 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (11287 ms) 2024-05-15T12:47:36.194 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2024-05-15T12:47:36.194 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (3466 ms) 2024-05-15T12:47:36.195 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2024-05-15T12:47:36.195 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (1140 ms) 2024-05-15T12:47:36.195 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2024-05-15T12:47:36.195 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (846 ms) 2024-05-15T12:47:36.195 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2024-05-15T12:47:36.195 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (2717 ms) 2024-05-15T12:47:36.195 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2024-05-15T12:47:36.195 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (2675 ms) 2024-05-15T12:47:36.195 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2024-05-15T12:47:36.195 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (258 ms) 2024-05-15T12:47:36.195 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2024-05-15T12:47:36.195 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (972 ms) 2024-05-15T12:47:36.195 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2024-05-15T12:47:36.195 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (2672 ms) 2024-05-15T12:47:36.195 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [----------] 10 tests from LibRadosIoEC (80358 ms total) 2024-05-15T12:47:36.196 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: 2024-05-15T12:47:36.196 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [----------] Global test environment tear-down 2024-05-15T12:47:36.196 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [==========] 24 tests from 2 test suites ran. (170625 ms total) 2024-05-15T12:47:36.196 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ PASSED ] 24 tests. 2024-05-15T12:47:42.391 INFO:tasks.workunit.client.0.smithi117.stdout:********* 2024-05-15T12:47:42.391 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_9_[74318]: starting. 2024-05-15T12:47:42.391 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_9_[74318]: creating pool ceph_test_rados_list_parallel.smithi117-40669 2024-05-15T12:47:42.391 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_9_[74318]: created object 0... 2024-05-15T12:47:42.391 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_9_[74318]: created object 25... 2024-05-15T12:47:42.391 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_9_[74318]: created object 49... 2024-05-15T12:47:42.391 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_9_[74318]: finishing. 2024-05-15T12:47:42.391 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_9_[74318]: shutting down. 2024-05-15T12:47:42.392 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-05-15T12:47:42.392 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-05-15T12:47:42.392 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-05-15T12:47:42.392 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_10_[74319]: starting. 2024-05-15T12:47:42.392 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_10_[74319]: listing objects. 2024-05-15T12:47:42.392 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_10_[74319]: listed object 0... 2024-05-15T12:47:42.392 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_10_[74319]: listed object 25... 2024-05-15T12:47:42.392 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_10_[74319]: listed object 50... 2024-05-15T12:47:42.392 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_10_[74319]: listed object 75... 2024-05-15T12:47:42.392 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_10_[74319]: saw 98 objects 2024-05-15T12:47:42.392 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_10_[74319]: shutting down. 2024-05-15T12:47:42.392 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-05-15T12:47:42.392 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-05-15T12:47:42.392 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-05-15T12:47:42.392 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_12_[74321]: starting. 2024-05-15T12:47:42.393 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_12_[74321]: added 25 objects... 2024-05-15T12:47:42.393 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_12_[74321]: added half of the objects 2024-05-15T12:47:42.393 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_12_[74321]: added 50 objects... 2024-05-15T12:47:42.393 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_12_[74321]: added 50 objects 2024-05-15T12:47:42.393 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_12_[74321]: shutting down. 2024-05-15T12:47:42.393 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-05-15T12:47:42.393 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-05-15T12:47:42.393 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-05-15T12:47:42.393 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_11_[74320]: starting. 2024-05-15T12:47:42.393 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_11_[74320]: added 25 objects... 2024-05-15T12:47:42.393 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_11_[74320]: added half of the objects 2024-05-15T12:47:42.393 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_11_[74320]: added 50 objects... 2024-05-15T12:47:42.393 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_11_[74320]: added 50 objects 2024-05-15T12:47:42.393 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_11_[74320]: shutting down. 2024-05-15T12:47:42.393 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-05-15T12:47:42.394 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-05-15T12:47:42.394 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-05-15T12:47:42.394 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_13_[74322]: starting. 2024-05-15T12:47:42.394 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_13_[74322]: removed 25 objects... 2024-05-15T12:47:42.394 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_13_[74322]: removed half of the objects 2024-05-15T12:47:42.394 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_13_[74322]: removed 50 objects... 2024-05-15T12:47:42.394 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_13_[74322]: removed 50 objects 2024-05-15T12:47:42.394 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_13_[74322]: shutting down. 2024-05-15T12:47:42.394 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-05-15T12:47:42.394 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-05-15T12:47:42.394 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-05-15T12:47:42.394 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-05-15T12:47:42.394 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_14_[85530]: starting. 2024-05-15T12:47:42.394 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_14_[85530]: creating pool ceph_test_rados_list_parallel.smithi117-40669 2024-05-15T12:47:42.394 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_14_[85530]: created object 0... 2024-05-15T12:47:42.395 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_14_[85530]: created object 25... 2024-05-15T12:47:42.395 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_14_[85530]: created object 49... 2024-05-15T12:47:42.395 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_14_[85530]: finishing. 2024-05-15T12:47:42.395 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_14_[85530]: shutting down. 2024-05-15T12:47:42.395 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-05-15T12:47:42.395 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-05-15T12:47:42.395 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-05-15T12:47:42.395 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-05-15T12:47:42.395 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_15_[85531]: starting. 2024-05-15T12:47:42.395 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_15_[85531]: listing objects. 2024-05-15T12:47:42.395 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_15_[85531]: listed object 0... 2024-05-15T12:47:48.455 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_15_[85531]: listed object 2109 ms) 2024-05-15T12:47:48.456 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2024-05-15T12:47:48.456 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (2937 ms) 2024-05-15T12:47:48.456 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2024-05-15T12:47:48.456 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (762 ms) 2024-05-15T12:47:48.456 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2024-05-15T12:47:48.456 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (2088 ms) 2024-05-15T12:47:48.456 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2024-05-15T12:47:48.456 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (637 ms) 2024-05-15T12:47:48.456 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2024-05-15T12:47:48.456 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (817 ms) 2024-05-15T12:47:48.456 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2024-05-15T12:47:48.457 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (2159 ms) 2024-05-15T12:47:48.457 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2024-05-15T12:47:48.457 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (219 ms) 2024-05-15T12:47:48.457 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2024-05-15T12:47:48.457 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (902 ms) 2024-05-15T12:47:48.457 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2024-05-15T12:47:48.457 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (278 ms) 2024-05-15T12:47:48.457 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2024-05-15T12:47:48.457 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (1264 ms) 2024-05-15T12:47:48.457 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2024-05-15T12:47:48.457 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (1152 ms) 2024-05-15T12:47:48.457 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2024-05-15T12:47:48.457 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (376 ms) 2024-05-15T12:47:48.458 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (86790 ms total) 2024-05-15T12:47:48.458 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: 2024-05-15T12:47:48.458 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [----------] Global test environment tear-down 2024-05-15T12:47:48.458 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [==========] 37 tests from 2 test suites ran. (182881 ms total) 2024-05-15T12:47:48.458 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ PASSED ] 37 tests. 2024-05-15T12:47:48.688 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: Running main() from gmock_main.cc 2024-05-15T12:47:48.688 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2024-05-15T12:47:48.688 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [----------] Global test environment set-up. 2024-05-15T12:47:48.688 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [----------] 8 tests from LibRadosLock 2024-05-15T12:47:48.688 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2024-05-15T12:47:48.688 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (18324 ms) 2024-05-15T12:47:48.688 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2024-05-15T12:47:48.689 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ OK ] LibRadosLock.LockShared (20468 ms) 2024-05-15T12:47:48.689 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2024-05-15T12:47:48.689 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (1402 ms) 2024-05-15T12:47:48.689 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2024-05-15T12:47:48.689 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (1101 ms) 2024-05-15T12:47:48.689 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2024-05-15T12:47:48.689 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (356 ms) 2024-05-15T12:47:48.689 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2024-05-15T12:47:48.689 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ OK ] LibRadosLock.Unlock (2284 ms) 2024-05-15T12:47:48.689 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2024-05-15T12:47:48.689 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (175 ms) 2024-05-15T12:47:48.689 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2024-05-15T12:47:48.689 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (76 ms) 2024-05-15T12:47:48.689 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [----------] 8 tests from LibRadosLock (44187 ms total) 2024-05-15T12:47:48.690 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: 2024-05-15T12:47:48.690 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2024-05-15T12:47:48.690 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2024-05-15T12:47:48.690 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (52795 ms) 2024-05-15T12:47:48.690 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2024-05-15T12:47:48.690 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (2456 ms) 2024-05-15T12:47:48.690 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2024-05-15T12:47:48.690 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (19140 ms) 2024-05-15T12:47:48.690 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2024-05-15T12:47:48.690 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (5017 ms) 2024-05-15T12:47:48.690 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2024-05-15T12:47:48.690 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (1464 ms) 2024-05-15T12:47:48.690 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2024-05-15T12:47:48.691 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (440 ms) 2024-05-15T12:47:48.691 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2024-05-15T12:47:48.691 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (1436 ms) 2024-05-15T12:47:48.691 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2024-05-15T12:47:48.691 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (801 ms) 2024-05-15T12:47:48.691 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (83550 ms total) 2024-05-15T12:47:48.691 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: 2024-05-15T12:47:48.691 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [----------] Global test environment tear-down 2024-05-15T12:47:48.691 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (183086 ms total) 2024-05-15T12:47:48.691 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ PASSED ] 16 tests. 2024-05-15T12:47:53.565 INFO:tasks.workunit.client.0.smithi117.stdout:5... 2024-05-15T12:47:53.565 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_15_[85531]: listed object 50... 2024-05-15T12:47:53.565 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_15_[85531]: listed object 75... 2024-05-15T12:47:53.565 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_15_[85531]: listed object 100... 2024-05-15T12:47:53.565 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_15_[85531]: listed object 125... 2024-05-15T12:47:53.565 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_15_[85531]: saw 150 objects 2024-05-15T12:47:53.565 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_15_[85531]: shutting down. 2024-05-15T12:47:53.565 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-05-15T12:47:53.565 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-05-15T12:47:53.565 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-05-15T12:47:53.565 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-05-15T12:47:53.565 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_16_[85532]: starting. 2024-05-15T12:47:53.565 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_16_[85532]: added 25 objects... 2024-05-15T12:47:53.566 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_16_[85532]: added half of the objects 2024-05-15T12:47:53.566 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_16_[85532]: added 50 objects... 2024-05-15T12:47:53.566 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_16_[85532]: added 50 objects 2024-05-15T12:47:53.566 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_16_[85532]: shutting down. 2024-05-15T12:47:53.566 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-05-15T12:47:53.566 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-05-15T12:47:53.566 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-05-15T12:47:53.566 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-05-15T12:47:53.566 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-05-15T12:47:53.566 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******* SUCCESS ********** 2024-05-15T12:47:58.335 INFO:tasks.workunit.client.0.smithi117.stdout:fy_id 485331304453 notifier_gid 4807 2024-05-15T12:47:58.336 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: timed out 2024-05-15T12:47:58.336 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: flushing 2024-05-15T12:47:58.336 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: flushed 2024-05-15T12:47:58.336 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3175 ms) 2024-05-15T12:47:58.336 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2024-05-15T12:47:58.336 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: trying... 2024-05-15T12:47:58.336 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: handle_notify cookie 94789780204912 notify_id 489626271747 notifier_gid 4807 2024-05-15T12:47:58.336 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: timed out 2024-05-15T12:47:58.336 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: flushing 2024-05-15T12:47:58.336 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: flushed 2024-05-15T12:47:58.336 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3336 ms) 2024-05-15T12:47:58.337 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2024-05-15T12:47:58.337 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: List watches 2024-05-15T12:47:58.337 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: notify2 2024-05-15T12:47:58.337 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: handle_notify cookie 94789780204912 notify_id 493921239046 notifier_gid 4807 2024-05-15T12:47:58.337 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: notify2 done 2024-05-15T12:47:58.337 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: watch_check 2024-05-15T12:47:58.337 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: unwatch2 2024-05-15T12:47:58.337 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: flushing 2024-05-15T12:47:58.337 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: done 2024-05-15T12:47:58.337 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3884 ms) 2024-05-15T12:47:58.337 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2024-05-15T12:47:58.337 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: List watches 2024-05-15T12:47:58.337 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: notify2 2024-05-15T12:47:58.337 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: handle_notify cookie 94789780204912 notify_id 498216206343 notifier_gid 4807 2024-05-15T12:47:58.337 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: notify2 done 2024-05-15T12:47:58.337 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: watch_check 2024-05-15T12:47:58.337 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: unwatch2 2024-05-15T12:47:58.338 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: flushing 2024-05-15T12:47:58.338 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: done 2024-05-15T12:47:58.338 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3510 ms) 2024-05-15T12:47:58.338 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (98532 ms total) 2024-05-15T12:47:58.338 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: 2024-05-15T12:47:58.338 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2024-05-15T12:47:58.338 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (192580 ms total) 2024-05-15T12:47:58.338 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2024-05-15T12:47:59.305 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd_pp: Running main() from gmock_main.cc 2024-05-15T12:47:59.305 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2024-05-15T12:47:59.305 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd_pp: [----------] Global test environment set-up. 2024-05-15T12:47:59.305 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2024-05-15T12:47:59.305 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2024-05-15T12:47:59.305 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (574 ms) 2024-05-15T12:47:59.306 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2024-05-15T12:47:59.306 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (440 ms) 2024-05-15T12:47:59.306 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2024-05-15T12:47:59.306 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (192475 ms) 2024-05-15T12:47:59.306 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (193489 ms total) 2024-05-15T12:47:59.306 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd_pp: 2024-05-15T12:47:59.306 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd_pp: [----------] Global test environment tear-down 2024-05-15T12:47:59.306 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (193489 ms total) 2024-05-15T12:47:59.306 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2024-05-15T12:48:00.641 INFO:tasks.workunit.client.0.smithi117.stdout:46fd542740 2024-05-15T12:48:00.641 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: waiting 0x5646fd540e70 2024-05-15T12:48:00.641 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: waiting 0x5646fd5432d0 2024-05-15T12:48:00.641 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: waiting 0x5646fd543dc0 2024-05-15T12:48:00.642 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: waiting 0x5646fd544060 2024-05-15T12:48:00.642 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: waiting 0x5646fd542450 2024-05-15T12:48:00.642 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: waiting 0x5646fcc9c600 2024-05-15T12:48:00.642 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: waiting 0x5646fd543ab0 2024-05-15T12:48:00.642 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: waiting 0x5646fcc9d140 2024-05-15T12:48:00.642 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: waiting 0x5646fcc9dbd0 2024-05-15T12:48:00.642 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: waiting 0x5646fcc9c300 2024-05-15T12:48:00.642 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: waiting 0x5646fcc9e730 2024-05-15T12:48:00.642 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: waiting 0x5646fcc9f1c0 2024-05-15T12:48:00.642 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: waiting 0x5646fcc9d930 2024-05-15T12:48:00.642 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: done. 2024-05-15T12:48:00.642 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (50658 ms) 2024-05-15T12:48:00.642 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (50658 ms total) 2024-05-15T12:48:00.642 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: 2024-05-15T12:48:00.642 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2024-05-15T12:48:00.643 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2024-05-15T12:48:00.643 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2024-05-15T12:48:00.643 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2024-05-15T12:48:00.643 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [ OK ] LibRadosMisc.Exec (38633 ms) 2024-05-15T12:48:00.643 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2024-05-15T12:48:00.643 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (7994 ms) 2024-05-15T12:48:00.643 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2024-05-15T12:48:00.643 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (7305 ms) 2024-05-15T12:48:00.643 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2024-05-15T12:48:00.643 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [ OK ] LibRadosMisc.Applications (5757 ms) 2024-05-15T12:48:00.643 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2024-05-15T12:48:00.643 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2024-05-15T12:48:00.643 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2024-05-15T12:48:00.643 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2024-05-15T12:48:00.643 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2024-05-15T12:48:00.643 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (76208 ms) 2024-05-15T12:48:00.643 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [----------] 8 tests from LibRadosMisc (135897 ms total) 2024-05-15T12:48:00.644 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: 2024-05-15T12:48:00.644 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [----------] Global test environment tear-down 2024-05-15T12:48:00.644 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (194971 ms total) 2024-05-15T12:48:00.644 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [ PASSED ] 12 tests. 2024-05-15T12:48:03.079 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:06:10.257 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:03.093 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:06:10.271 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:03.102 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:06:10.280 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:03.105 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:06:10.283 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:03.165 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: Running main() from gmock_main.cc 2024-05-15T12:48:03.165 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2024-05-15T12:48:03.166 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [----------] Global test environment set-up. 2024-05-15T12:48:03.166 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2024-05-15T12:48:03.166 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2024-05-15T12:48:03.166 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: watch_notify_test_cb 2024-05-15T12:48:03.166 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (37397 ms) 2024-05-15T12:48:03.166 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2024-05-15T12:48:03.166 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-05-15T12:48:03.166 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94780436672672 err -107 2024-05-15T12:48:03.166 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (3292 ms) 2024-05-15T12:48:03.166 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2024-05-15T12:48:03.166 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-05-15T12:48:03.166 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94780436684512 err -107 2024-05-15T12:48:03.166 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (2212 ms) 2024-05-15T12:48:03.167 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2024-05-15T12:48:03.167 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: watch_notify2_test_cb from 4442 notify_id 210453397504 cookie 94780436684512 2024-05-15T12:48:03.167 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (921 ms) 2024-05-15T12:48:03.167 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2024-05-15T12:48:03.167 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: watch_notify2_test_cb from 4442 notify_id 210453397504 cookie 94780436725808 2024-05-15T12:48:03.167 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (7461 ms) 2024-05-15T12:48:03.167 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2024-05-15T12:48:03.167 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: watch_notify2_test_cb from 4442 notify_id 326417514497 cookie 94780436725808 2024-05-15T12:48:03.167 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (38585 ms) 2024-05-15T12:48:03.167 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2024-05-15T12:48:03.167 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: watch_notify2_test_cb from 4442 notify_id 459561500672 cookie 94780436741232 2024-05-15T12:48:03.167 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: watch_notify2_test_cb from 4442 notify_id 459561500672 cookie 94780436742288 2024-05-15T12:48:03.167 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: watch_notify2_test_cb from 4442 notify_id 463856467969 cookie 94780436741232 2024-05-15T12:48:03.167 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: watch_notify2_test_cb from 4442 notify_id 463856467969 cookie 94780436742288 2024-05-15T12:48:03.168 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (33470 ms) 2024-05-15T12:48:03.168 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2024-05-15T12:48:03.168 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: watch_notify2_test_cb from 4442 notify_id 468151435265 cookie 94780436757744 2024-05-15T12:48:03.168 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: watch_notify2_test_cb from 4442 notify_id 468151435266 cookie 94780436757744 2024-05-15T12:48:03.168 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: watch_notify2_test_cb from 4442 notify_id 468151435266 cookie 94780436757744 2024-05-15T12:48:03.168 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: watch_notify2_test_cb from 4442 notify_id 468151435267 cookie 94780436757744 2024-05-15T12:48:03.168 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3193 ms) 2024-05-15T12:48:03.168 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2024-05-15T12:48:03.168 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2024-05-15T12:48:03.168 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94780436743520 err -107 2024-05-15T12:48:03.168 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: watch_notify2_test_cb from 4442 notify_id 485331304451 cookie 94780436743520 2024-05-15T12:48:03.168 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (10366 ms) 2024-05-15T12:48:03.168 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2024-05-15T12:48:03.169 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94780436743520 err -107 2024-05-15T12:48:03.169 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2024-05-15T12:48:03.169 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (296 ms) 2024-05-15T12:48:03.169 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (137193 ms total) 2024-05-15T12:48:03.169 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: 2024-05-15T12:48:03.169 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2024-05-15T12:48:03.169 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2024-05-15T12:48:03.173 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:06:10.344 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:03.174 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:06:10.346 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:03.269 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:06:10.447 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:03.337 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:06:10.516 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:03.362 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:06:10.534 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:07.867 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: watch_notify_test_cb 2024-05-15T12:48:07.867 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (9864 ms) 2024-05-15T12:48:07.868 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (9864 ms total) 2024-05-15T12:48:07.868 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: 2024-05-15T12:48:07.868 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [----------] Global test environment tear-down 2024-05-15T12:48:07.868 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (202119 ms total) 2024-05-15T12:48:07.868 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [ PASSED ] 11 tests. 2024-05-15T12:48:23.230 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:06:30.408 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:23.291 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:06:30.469 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:23.361 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:06:30.532 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:23.362 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:06:30.534 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:23.366 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:06:30.544 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:23.370 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:06:30.549 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:23.393 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:06:30.571 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:23.433 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:06:30.605 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:23.450 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:06:30.628 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:43.487 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:06:50.665 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:43.510 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:06:50.688 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:43.519 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:06:50.690 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:43.522 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:06:50.694 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:43.525 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:06:50.703 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:43.526 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:06:50.705 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:43.630 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:06:50.808 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:43.714 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:06:50.885 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:43.784 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:06:50.962 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:52.707 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: Running main() from gmock_main.cc 2024-05-15T12:48:52.707 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [==========] Running 14 tests from 1 test suite. 2024-05-15T12:48:52.707 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [----------] Global test environment set-up. 2024-05-15T12:48:52.707 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps 2024-05-15T12:48:52.707 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ RUN ] NeoRadosReadOps.SetOpFlags 2024-05-15T12:48:52.707 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ OK ] NeoRadosReadOps.SetOpFlags (32779 ms) 2024-05-15T12:48:52.707 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertExists 2024-05-15T12:48:52.707 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertExists (57134 ms) 2024-05-15T12:48:52.707 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertVersion 2024-05-15T12:48:52.707 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertVersion (27179 ms) 2024-05-15T12:48:52.707 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpXattr 2024-05-15T12:48:52.707 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpXattr (29698 ms) 2024-05-15T12:48:52.707 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ RUN ] NeoRadosReadOps.Read 2024-05-15T12:48:52.708 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ OK ] NeoRadosReadOps.Read (37459 ms) 2024-05-15T12:48:52.708 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ RUN ] NeoRadosReadOps.Checksum 2024-05-15T12:48:52.708 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ OK ] NeoRadosReadOps.Checksum (15420 ms) 2024-05-15T12:48:52.708 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ RUN ] NeoRadosReadOps.RWOrderedRead 2024-05-15T12:48:52.708 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ OK ] NeoRadosReadOps.RWOrderedRead (7318 ms) 2024-05-15T12:48:52.708 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ RUN ] NeoRadosReadOps.ShortRead 2024-05-15T12:48:52.708 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ OK ] NeoRadosReadOps.ShortRead (5095 ms) 2024-05-15T12:48:52.708 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ RUN ] NeoRadosReadOps.Exec 2024-05-15T12:48:52.708 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ OK ] NeoRadosReadOps.Exec (4300 ms) 2024-05-15T12:48:52.708 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ RUN ] NeoRadosReadOps.Stat 2024-05-15T12:48:52.708 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ OK ] NeoRadosReadOps.Stat (5249 ms) 2024-05-15T12:48:52.708 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ RUN ] NeoRadosReadOps.Omap 2024-05-15T12:48:52.708 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ OK ] NeoRadosReadOps.Omap (4139 ms) 2024-05-15T12:48:52.708 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ RUN ] NeoRadosReadOps.OmapNuls 2024-05-15T12:48:52.708 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ OK ] NeoRadosReadOps.OmapNuls (6964 ms) 2024-05-15T12:48:52.709 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ RUN ] NeoRadosReadOps.GetXattrs 2024-05-15T12:48:52.709 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ OK ] NeoRadosReadOps.GetXattrs (7399 ms) 2024-05-15T12:48:52.709 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpExt 2024-05-15T12:48:52.709 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpExt (6492 ms) 2024-05-15T12:48:52.709 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps (246625 ms total) 2024-05-15T12:48:52.709 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: 2024-05-15T12:48:52.709 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [----------] Global test environment tear-down 2024-05-15T12:48:52.709 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [==========] 14 tests from 1 test suite ran. (246625 ms total) 2024-05-15T12:48:52.709 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ PASSED ] 14 tests. 2024-05-15T12:48:56.154 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: Running main() from gmock_main.cc 2024-05-15T12:48:56.154 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [==========] Running 21 tests from 5 test suites. 2024-05-15T12:48:56.154 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2024-05-15T12:48:56.154 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2024-05-15T12:48:56.154 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: seed 40261 2024-05-15T12:48:56.154 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2024-05-15T12:48:56.155 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (28044 ms) 2024-05-15T12:48:56.155 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2024-05-15T12:48:56.155 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (22059 ms) 2024-05-15T12:48:56.155 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2024-05-15T12:48:56.155 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (31382 ms) 2024-05-15T12:48:56.155 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2024-05-15T12:48:56.155 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (7320 ms) 2024-05-15T12:48:56.155 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2024-05-15T12:48:56.155 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (9448 ms) 2024-05-15T12:48:56.155 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (98253 ms total) 2024-05-15T12:48:56.155 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: 2024-05-15T12:48:56.155 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2024-05-15T12:48:56.155 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2024-05-15T12:48:56.155 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (15510 ms) 2024-05-15T12:48:56.155 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2024-05-15T12:48:56.156 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (38116 ms) 2024-05-15T12:48:56.156 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2024-05-15T12:48:56.156 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (36509 ms) 2024-05-15T12:48:56.156 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2024-05-15T12:48:56.156 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (10587 ms) 2024-05-15T12:48:56.156 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2024-05-15T12:48:56.156 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (4879 ms) 2024-05-15T12:48:56.156 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-05-15T12:48:56.156 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1882-gae248125/rpm/el9/BUILD/ceph-19.0.0-1882-gae248125/src/test/librados/snapshots_cxx.cc:463: Skipped 2024-05-15T12:48:56.156 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: 2024-05-15T12:48:56.156 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2024-05-15T12:48:56.156 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2024-05-15T12:48:56.156 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: deleting snap 14 in pool LibRadosSnapshotsSelfManagedPP_smithi117-40261-7 2024-05-15T12:48:56.156 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: waiting for snaps to purge 2024-05-15T12:48:56.157 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (21646 ms) 2024-05-15T12:48:56.157 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (127247 ms total) 2024-05-15T12:48:56.157 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: 2024-05-15T12:48:56.157 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode 2024-05-15T12:48:56.157 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected 2024-05-15T12:48:56.157 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected (12 ms) 2024-05-15T12:48:56.157 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance 2024-05-15T12:48:56.157 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance (6503 ms) 2024-05-15T12:48:56.157 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode (6515 ms total) 2024-05-15T12:48:56.157 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: 2024-05-15T12:48:56.157 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2024-05-15T12:48:56.157 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2024-05-15T12:48:56.157 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (5369 ms) 2024-05-15T12:48:56.157 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2024-05-15T12:49:03.530 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:07:10.708 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:03.536 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:07:10.714 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:03.609 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:07:10.788 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:03.611 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:07:10.782 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:03.612 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:07:10.784 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:03.614 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:07:10.793 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:03.630 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:07:10.809 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:03.690 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:07:10.861 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:03.700 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:07:10.878 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:07.502 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ OK ] io: Running main() from gmock_main.cc 2024-05-15T12:49:07.776 INFO:tasks.workunit.client.0.smithi117.stdout: io: [==========] Running 14 tests from 1 test suite. 2024-05-15T12:49:07.777 INFO:tasks.workunit.client.0.smithi117.stdout: io: [----------] Global test environment set-up. 2024-05-15T12:49:07.777 INFO:tasks.workunit.client.0.smithi117.stdout: io: [----------] 14 tests from NeoRadosIo 2024-05-15T12:49:07.777 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ RUN ] NeoRadosIo.Limits 2024-05-15T12:49:07.777 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ OK ] NeoRadosIo.Limits (50178 ms) 2024-05-15T12:49:07.777 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ RUN ] NeoRadosIo.SimpleWrite 2024-05-15T12:49:07.777 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ OK ] NeoRadosIo.SimpleWrite (56510 ms) 2024-05-15T12:49:07.777 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ RUN ] NeoRadosIo.ReadOp 2024-05-15T12:49:07.777 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ OK ] NeoRadosIo.ReadOp (29208 ms) 2024-05-15T12:49:07.777 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ RUN ] NeoRadosIo.SparseRead 2024-05-15T12:49:07.777 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ OK ] NeoRadosIo.SparseRead (36051 ms) 2024-05-15T12:49:07.777 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ RUN ] NeoRadosIo.RoundTrip 2024-05-15T12:49:07.777 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ OK ] NeoRadosIo.RoundTrip (22643 ms) 2024-05-15T12:49:07.778 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ RUN ] NeoRadosIo.ReadIntoBuufferlist 2024-05-15T12:49:07.778 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ OK ] NeoRadosIo.ReadIntoBuufferlist (6147 ms) 2024-05-15T12:49:07.778 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ RUN ] NeoRadosIo.OverlappingWriteRoundTrip 2024-05-15T12:49:07.778 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ OK ] NeoRadosIo.OverlappingWriteRoundTrip (10617 ms) 2024-05-15T12:49:07.778 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ RUN ] NeoRadosIo.WriteFullRoundTrip 2024-05-15T12:49:07.778 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ OK ] NeoRadosIo.WriteFullRoundTrip (5856 ms) 2024-05-15T12:49:07.778 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ RUN ] NeoRadosIo.AppendRoundTrip 2024-05-15T12:49:07.778 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ OK ] NeoRadosIo.AppendRoundTrip (6547 ms) 2024-05-15T12:49:07.778 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ RUN ] NeoRadosIo.Trunc 2024-05-15T12:49:07.778 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ OK ] NeoRadosIo.Trunc (5844 ms) 2024-05-15T12:49:07.778 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ RUN ] NeoRadosIo.Remove 2024-05-15T12:49:07.778 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ OK ] NeoRadosIo.Remove (5295 ms) 2024-05-15T12:49:07.778 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ RUN ] NeoRadosIo.XattrsRoundTrip 2024-05-15T12:49:07.778 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ OK ] NeoRadosIo.XattrsRoundTrip (5254 ms) 2024-05-15T12:49:07.779 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ RUN ] NeoRadosIo.RmXattr 2024-05-15T12:49:07.779 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ OK ] NeoRadosIo.RmXattr (6525 ms) 2024-05-15T12:49:07.779 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ RUN ] NeoRadosIo.GetXattrs 2024-05-15T12:49:07.779 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ OK ] NeoRadosIo.GetXattrs (14807 ms) 2024-05-15T12:49:07.779 INFO:tasks.workunit.client.0.smithi117.stdout: io: [----------] 14 tests from NeoRadosIo (261482 ms total) 2024-05-15T12:49:07.779 INFO:tasks.workunit.client.0.smithi117.stdout: io: 2024-05-15T12:49:07.779 INFO:tasks.workunit.client.0.smithi117.stdout: io: [----------] Global test environment tear-down 2024-05-15T12:49:07.779 INFO:tasks.workunit.client.0.smithi117.stdout: io: [==========] 14 tests from 1 test suite ran. (261482 ms total) 2024-05-15T12:49:07.779 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ PASSED ] 14 tests. 2024-05-15T12:49:13.546 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: Running main() from gmock_main.cc 2024-05-15T12:49:13.546 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2024-05-15T12:49:13.546 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [----------] Global test environment set-up. 2024-05-15T12:49:13.546 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2024-05-15T12:49:13.546 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2024-05-15T12:49:13.546 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (23622 ms) 2024-05-15T12:49:13.546 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2024-05-15T12:49:13.546 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (23706 ms) 2024-05-15T12:49:13.546 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2024-05-15T12:49:13.547 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (39894 ms) 2024-05-15T12:49:13.547 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2024-05-15T12:49:13.547 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (26432 ms) 2024-05-15T12:49:13.547 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (113654 ms total) 2024-05-15T12:49:13.547 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: 2024-05-15T12:49:13.547 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2024-05-15T12:49:13.547 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2024-05-15T12:49:13.547 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (46157 ms) 2024-05-15T12:49:13.547 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2024-05-15T12:49:13.547 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (14971 ms) 2024-05-15T12:49:13.547 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2024-05-15T12:49:13.547 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (7859 ms) 2024-05-15T12:49:13.547 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (68987 ms total) 2024-05-15T12:49:13.547 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: 2024-05-15T12:49:13.547 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2024-05-15T12:49:13.547 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2024-05-15T12:49:13.548 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (6409 ms) 2024-05-15T12:49:13.548 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2024-05-15T12:49:13.548 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (2599 ms) 2024-05-15T12:49:13.548 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2024-05-15T12:49:13.548 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (5324 ms) 2024-05-15T12:49:13.548 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2024-05-15T12:49:13.548 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (3206 ms) 2024-05-15T12:49:13.548 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (17538 ms total) 2024-05-15T12:49:13.548 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: 2024-05-15T12:49:13.548 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2024-05-15T12:49:13.548 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2024-05-15T12:49:13.548 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (17056 ms) 2024-05-15T12:49:13.548 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2024-05-15T12:49:13.548 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (5650 ms) 2024-05-15T12:49:13.548 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (22706 ms total) 2024-05-15T12:49:13.549 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: 2024-05-15T12:49:13.549 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [----------] Global test environment tear-down 2024-05-15T12:49:13.549 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (267862 ms total) 2024-05-15T12:49:13.549 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [ PASSED ] 13 tests. 2024-05-15T12:49:19.597 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: Running main() from gmock_main.cc 2024-05-15T12:49:19.597 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [==========] Running 11 tests from 2 test suites. 2024-05-15T12:49:19.597 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [----------] Global test environment set-up. 2024-05-15T12:49:19.597 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots 2024-05-15T12:49:19.598 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapList 2024-05-15T12:49:19.598 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapList (53059 ms) 2024-05-15T12:49:19.598 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapRemove 2024-05-15T12:49:19.598 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapRemove (50291 ms) 2024-05-15T12:49:19.598 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [ RUN ] NeoRadosSnapshots.Rollback 2024-05-15T12:49:19.598 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [ OK ] NeoRadosSnapshots.Rollback (13379 ms) 2024-05-15T12:49:19.598 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapGetName 2024-05-15T12:49:19.598 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapGetName (48471 ms) 2024-05-15T12:49:19.598 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapCreateRemove 2024-05-15T12:49:19.598 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapCreateRemove (36578 ms) 2024-05-15T12:49:19.598 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots (201778 ms total) 2024-05-15T12:49:19.598 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: 2024-05-15T12:49:19.598 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps 2024-05-15T12:49:19.598 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Snap 2024-05-15T12:49:19.598 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Snap (8302 ms) 2024-05-15T12:49:19.599 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Rollback 2024-05-15T12:49:19.599 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Rollback (9214 ms) 2024-05-15T12:49:19.599 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.SnapOverlap 2024-05-15T12:49:19.599 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.SnapOverlap (20813 ms) 2024-05-15T12:49:19.599 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Bug11677 2024-05-15T12:49:19.599 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Bug11677 (8887 ms) 2024-05-15T12:49:19.599 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.OrderSnap 2024-05-15T12:49:19.599 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.OrderSnap (4564 ms) 2024-05-15T12:49:19.599 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.ReusePurgedSnap 2024-05-15T12:49:19.599 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: Deleting snap 3 in pool ReusePurgedSnapsmithi117-41114-11. 2024-05-15T12:49:19.599 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: Waiting for snaps to purge. 2024-05-15T12:49:19.599 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.ReusePurgedSnap (19951 ms) 2024-05-15T12:49:19.599 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps (71731 ms total) 2024-05-15T12:49:19.599 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: 2024-05-15T12:49:19.600 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [----------] Global test environment tear-down 2024-05-15T12:49:19.600 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [==========] 11 tests from 2 test suites ran. (273509 ms total) 2024-05-15T12:49:19.600 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [ PASSED ] 11 tests. 2024-05-15T12:49:23.536 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:07:30.715 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:23.540 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:07:30.718 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:23.596 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:07:30.767 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:23.598 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:07:30.770 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:23.601 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:07:30.779 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:23.608 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:07:30.787 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:23.611 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:07:30.789 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:23.655 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:07:30.826 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:23.663 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:07:30.841 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:38.694 INFO:tasks.workunit.client.0.smithi117.stdout:LibRadosSnapshotsECPP.SnapRemovePP (2148 ms) 2024-05-15T12:49:38.695 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2024-05-15T12:49:38.695 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (11324 ms) 2024-05-15T12:49:38.695 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2024-05-15T12:49:38.695 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (3075 ms) 2024-05-15T12:49:38.695 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (21916 ms total) 2024-05-15T12:49:38.695 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: 2024-05-15T12:49:38.695 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2024-05-15T12:49:38.695 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2024-05-15T12:49:38.695 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (6578 ms) 2024-05-15T12:49:38.695 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2024-05-15T12:49:38.695 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (4031 ms) 2024-05-15T12:49:38.695 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2024-05-15T12:49:38.695 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (8272 ms) 2024-05-15T12:49:38.696 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (18881 ms total) 2024-05-15T12:49:38.696 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: 2024-05-15T12:49:38.696 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2024-05-15T12:49:38.696 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [==========] 21 tests from 5 test suites ran. (293000 ms total) 2024-05-15T12:49:38.696 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ PASSED ] 20 tests. 2024-05-15T12:49:38.696 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2024-05-15T12:49:38.696 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-05-15T12:49:43.837 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:07:51.016 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:43.843 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:07:51.021 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:43.909 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:07:51.087 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:43.913 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:07:51.091 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:43.914 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:07:51.093 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:43.915 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:07:51.086 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:43.942 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:07:51.113 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:43.973 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:07:51.152 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:44.017 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:07:51.186 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:56.646 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : cursor()=5:863748b0:::15:head expected=5:863748b0:::15:head 2024-05-15T12:49:56.647 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 5:863748b0:::15:head -> 15 2024-05-15T12:49:56.647 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : entry=15 expected=15 2024-05-15T12:49:56.647 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 5:cfc208b3:::3:head 2024-05-15T12:49:56.647 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : cursor()=5:cfc208b3:::3:head expected=5:cfc208b3:::3:head 2024-05-15T12:49:56.647 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 5:cfc208b3:::3:head -> 3 2024-05-15T12:49:56.647 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : entry=3 expected=3 2024-05-15T12:49:56.647 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 5:c4fdafeb:::6:head 2024-05-15T12:49:56.647 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : cursor()=5:c4fdafeb:::6:head expected=5:c4fdafeb:::6:head 2024-05-15T12:49:56.647 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 5:c4fdafeb:::6:head -> 6 2024-05-15T12:49:56.647 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : entry=6 expected=6 2024-05-15T12:49:56.647 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 5:bd63b0f1:::8:head 2024-05-15T12:49:56.647 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : cursor()=5:bd63b0f1:::8:head expected=5:bd63b0f1:::8:head 2024-05-15T12:49:56.647 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 5:bd63b0f1:::8:head -> 8 2024-05-15T12:49:56.647 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : entry=8 expected=8 2024-05-15T12:49:56.647 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 5:b29083e3:::5:head 2024-05-15T12:49:56.648 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : cursor()=5:b29083e3:::5:head expected=5:b29083e3:::5:head 2024-05-15T12:49:56.648 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 5:b29083e3:::5:head -> 5 2024-05-15T12:49:56.648 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : entry=5 expected=5 2024-05-15T12:49:56.648 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 5:89d3ae78:::11:head 2024-05-15T12:49:56.648 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : cursor()=5:89d3ae78:::11:head expected=5:89d3ae78:::11:head 2024-05-15T12:49:56.648 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 5:89d3ae78:::11:head -> 11 2024-05-15T12:49:56.648 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : entry=11 expected=11 2024-05-15T12:49:56.648 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 5:6cac518f:::0:head 2024-05-15T12:49:56.648 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : cursor()=5:6cac518f:::0:head expected=5:6cac518f:::0:head 2024-05-15T12:49:56.648 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 5:6cac518f:::0:head -> 0 2024-05-15T12:49:56.648 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : entry=0 expected=0 2024-05-15T12:49:56.648 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 5:f905c69b:::2:head 2024-05-15T12:49:56.648 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : cursor()=5:f905c69b:::2:head expected=5:f905c69b:::2:head 2024-05-15T12:49:56.648 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 5:f905c69b:::2:head -> 2 2024-05-15T12:49:56.648 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : entry=2 expected=2 2024-05-15T12:49:56.648 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (9062 ms) 2024-05-15T12:49:56.648 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2024-05-15T12:49:56.649 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (122311 ms) 2024-05-15T12:49:56.649 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2024-05-15T12:49:56.649 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: split 0/5 -> MIN 5:33333333::::head 2024-05-15T12:49:56.649 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: split 1/5 -> 5:33333333::::head 5:66666666::::head 2024-05-15T12:49:56.649 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: split 2/5 -> 5:66666666::::head 5:99999999::::head 2024-05-15T12:49:56.649 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: split 3/5 -> 5:99999999::::head 5:cccccccc::::head 2024-05-15T12:49:56.649 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: split 4/5 -> 5:cccccccc::::head MAX 2024-05-15T12:49:56.649 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (22951 ms) 2024-05-15T12:49:56.649 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [----------] 7 tests from LibRadosList (299342 ms total) 2024-05-15T12:49:56.649 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 2024-05-15T12:49:56.649 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [----------] 3 tests from LibRadosListEC 2024-05-15T12:49:56.649 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2024-05-15T12:49:56.649 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (1980 ms) 2024-05-15T12:49:56.649 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2024-05-15T12:49:56.650 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: myset foo1,foo2,foo3 2024-05-15T12:49:56.650 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: foo1 2024-05-15T12:49:56.650 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: foo2 2024-05-15T12:49:56.650 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: foo3 2024-05-15T12:49:56.650 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: myset foo1,foo4,foo5 2024-05-15T12:49:56.650 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: foo4 2024-05-15T12:49:56.650 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: foo5 2024-05-15T12:49:56.650 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: foo1 2024-05-15T12:49:56.650 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: myset foo6,foo7 2024-05-15T12:49:56.650 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: foo7 2024-05-15T12:49:56.650 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: foo6 2024-05-15T12:49:56.650 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-05-15T12:49:56.650 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: ns1:foo4 2024-05-15T12:49:56.650 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: ns1:foo5 2024-05-15T12:49:56.651 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: ns2:foo7 2024-05-15T12:49:56.651 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: ns2:foo6 2024-05-15T12:49:56.651 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: ns1:foo1 2024-05-15T12:49:56.651 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: :foo1 2024-05-15T12:49:56.651 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: :foo2 2024-05-15T12:49:56.651 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: :foo3 2024-05-15T12:49:56.651 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (1773 ms) 2024-05-15T12:49:56.651 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2024-05-15T12:49:56.651 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 1 0 2024-05-15T12:49:56.651 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 10 0 2024-05-15T12:49:56.651 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 13 0 2024-05-15T12:50:03.752 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:08:10.930 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:03.769 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:08:10.947 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:03.810 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:08:10.989 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:03.812 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:08:10.991 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:03.828 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:08:11.006 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:03.828 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:08:10.999 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:03.831 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:08:11.003 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:03.862 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:08:11.040 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:03.871 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:08:11.042 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:04.268 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 7 0 2024-05-15T12:50:04.268 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 14 0 2024-05-15T12:50:04.268 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 0 0 2024-05-15T12:50:04.269 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 15 0 2024-05-15T12:50:04.269 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 11 0 2024-05-15T12:50:04.269 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 5 0 2024-05-15T12:50:04.269 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 8 0 2024-05-15T12:50:04.269 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 6 0 2024-05-15T12:50:04.269 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 3 0 2024-05-15T12:50:04.269 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 4 0 2024-05-15T12:50:04.269 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 12 0 2024-05-15T12:50:04.269 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 9 0 2024-05-15T12:50:04.269 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 2 0 2024-05-15T12:50:04.269 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-05-15T12:50:04.269 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (2428 ms) 2024-05-15T12:50:04.269 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [----------] 3 tests from LibRadosListEC (6181 ms total) 2024-05-15T12:50:04.269 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 2024-05-15T12:50:04.269 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [----------] 1 test from LibRadosListNP 2024-05-15T12:50:04.269 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2024-05-15T12:50:04.270 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (3672 ms) 2024-05-15T12:50:04.270 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [----------] 1 test from LibRadosListNP (3672 ms total) 2024-05-15T12:50:04.270 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 2024-05-15T12:50:04.270 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [----------] Global test environment tear-down 2024-05-15T12:50:04.270 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [==========] 11 tests from 3 test suites ran. (318649 ms total) 2024-05-15T12:50:04.270 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [ PASSED ] 11 tests. 2024-05-15T12:50:23.920 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:08:31.098 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:23.982 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:08:31.160 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:23.992 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:08:31.163 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:23.997 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:08:31.168 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:24.021 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:08:31.199 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:24.038 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:08:31.209 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:24.081 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:08:31.259 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:24.082 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:08:31.260 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:24.116 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:08:31.294 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:27.090 INFO:tasks.workunit.client.0.smithi117.stdout: ec_list: Running main() from gmock_main.cc 2024-05-15T12:50:27.091 INFO:tasks.workunit.client.0.smithi117.stdout: ec_list: [==========] Running 3 tests from 1 test suite. 2024-05-15T12:50:27.091 INFO:tasks.workunit.client.0.smithi117.stdout: ec_list: [----------] Global test environment set-up. 2024-05-15T12:50:27.091 INFO:tasks.workunit.client.0.smithi117.stdout: ec_list: [----------] 3 tests from NeoradosECList 2024-05-15T12:50:27.091 INFO:tasks.workunit.client.0.smithi117.stdout: ec_list: [ RUN ] NeoradosECList.ListObjects 2024-05-15T12:50:27.091 INFO:tasks.workunit.client.0.smithi117.stdout: ec_list: [ OK ] NeoradosECList.ListObjects (54568 ms) 2024-05-15T12:50:27.091 INFO:tasks.workunit.client.0.smithi117.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsNS 2024-05-15T12:50:27.091 INFO:tasks.workunit.client.0.smithi117.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsNS (110464 ms) 2024-05-15T12:50:27.091 INFO:tasks.workunit.client.0.smithi117.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsMany 2024-05-15T12:50:27.091 INFO:tasks.workunit.client.0.smithi117.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsMany (176030 ms) 2024-05-15T12:50:27.091 INFO:tasks.workunit.client.0.smithi117.stdout: ec_list: [----------] 3 tests from NeoradosECList (341062 ms total) 2024-05-15T12:50:27.091 INFO:tasks.workunit.client.0.smithi117.stdout: ec_list: 2024-05-15T12:50:27.091 INFO:tasks.workunit.client.0.smithi117.stdout: ec_list: [----------] Global test environment tear-down 2024-05-15T12:50:27.091 INFO:tasks.workunit.client.0.smithi117.stdout: ec_list: [==========] 3 tests from 1 test suite ran. (341062 ms total) 2024-05-15T12:50:27.091 INFO:tasks.workunit.client.0.smithi117.stdout: ec_list: [ PASSED ] 3 tests. 2024-05-15T12:50:36.086 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_write_operations: Running main() from gmock_main.cc 2024-05-15T12:50:36.086 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2024-05-15T12:50:36.087 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_write_operations: [----------] Global test environment set-up. 2024-05-15T12:50:36.087 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2024-05-15T12:50:36.087 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2024-05-15T12:50:36.087 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2024-05-15T12:50:36.087 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2024-05-15T12:50:36.087 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_write_operations: 2024-05-15T12:50:36.087 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2024-05-15T12:50:36.087 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2024-05-15T12:50:36.087 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (17370 ms) 2024-05-15T12:50:36.087 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2024-05-15T12:50:36.087 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (30996 ms) 2024-05-15T12:50:36.087 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2024-05-15T12:50:36.087 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (67362 ms) 2024-05-15T12:50:36.087 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2024-05-15T12:50:36.087 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (20497 ms) 2024-05-15T12:50:36.088 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2024-05-15T12:50:36.088 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (36382 ms) 2024-05-15T12:50:36.088 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2024-05-15T12:50:36.088 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (21043 ms) 2024-05-15T12:50:36.088 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2024-05-15T12:50:36.088 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (156597 ms) 2024-05-15T12:50:36.088 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (350247 ms total) 2024-05-15T12:50:36.088 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_write_operations: 2024-05-15T12:50:36.088 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_write_operations: [----------] Global test environment tear-down 2024-05-15T12:50:36.088 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (350247 ms total) 2024-05-15T12:50:36.088 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2024-05-15T12:50:41.423 INFO:tasks.workunit.client.0.smithi117.stdout: write_operations: Running main() from gmock_main.cc 2024-05-15T12:50:41.423 INFO:tasks.workunit.client.0.smithi117.stdout: write_operations: [==========] Running 7 tests from 1 test suite. 2024-05-15T12:50:41.423 INFO:tasks.workunit.client.0.smithi117.stdout: write_operations: [----------] Global test environment set-up. 2024-05-15T12:50:41.423 INFO:tasks.workunit.client.0.smithi117.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps 2024-05-15T12:50:41.423 INFO:tasks.workunit.client.0.smithi117.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertExists 2024-05-15T12:50:41.423 INFO:tasks.workunit.client.0.smithi117.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertExists (50111 ms) 2024-05-15T12:50:41.424 INFO:tasks.workunit.client.0.smithi117.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertVersion 2024-05-15T12:50:41.424 INFO:tasks.workunit.client.0.smithi117.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertVersion (33002 ms) 2024-05-15T12:50:41.424 INFO:tasks.workunit.client.0.smithi117.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Xattrs 2024-05-15T12:50:41.424 INFO:tasks.workunit.client.0.smithi117.stdout: write_operations: [ OK ] NeoRadosWriteOps.Xattrs (14472 ms) 2024-05-15T12:50:41.424 INFO:tasks.workunit.client.0.smithi117.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Write 2024-05-15T12:50:41.424 INFO:tasks.workunit.client.0.smithi117.stdout: write_operations: [ OK ] NeoRadosWriteOps.Write (38852 ms) 2024-05-15T12:50:41.424 INFO:tasks.workunit.client.0.smithi117.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Exec 2024-05-15T12:50:41.424 INFO:tasks.workunit.client.0.smithi117.stdout: write_operations: [ OK ] NeoRadosWriteOps.Exec (44793 ms) 2024-05-15T12:50:41.424 INFO:tasks.workunit.client.0.smithi117.stdout: write_operations: [ RUN ] NeoRadosWriteOps.WriteSame 2024-05-15T12:50:41.424 INFO:tasks.workunit.client.0.smithi117.stdout: write_operations: [ OK ] NeoRadosWriteOps.WriteSame (17123 ms) 2024-05-15T12:50:41.424 INFO:tasks.workunit.client.0.smithi117.stdout: write_operations: [ RUN ] NeoRadosWriteOps.CmpExt 2024-05-15T12:50:41.424 INFO:tasks.workunit.client.0.smithi117.stdout: write_operations: [ OK ] NeoRadosWriteOps.CmpExt (156976 ms) 2024-05-15T12:50:41.424 INFO:tasks.workunit.client.0.smithi117.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps (355330 ms total) 2024-05-15T12:50:41.424 INFO:tasks.workunit.client.0.smithi117.stdout: write_operations: 2024-05-15T12:50:41.424 INFO:tasks.workunit.client.0.smithi117.stdout: write_operations: [----------] Global test environment tear-down 2024-05-15T12:50:41.425 INFO:tasks.workunit.client.0.smithi117.stdout: write_operations: [==========] 7 tests from 1 test suite ran. (355330 ms total) 2024-05-15T12:50:41.425 INFO:tasks.workunit.client.0.smithi117.stdout: write_operations: [ PASSED ] 7 tests. 2024-05-15T12:50:43.511 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:08:50.689 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:43.516 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:08:50.694 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:43.558 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:08:50.736 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:43.567 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:08:50.739 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:43.570 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:08:50.742 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:43.586 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:08:50.764 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:43.587 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:08:50.766 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:43.619 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:08:50.790 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:43.629 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:08:50.808 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:44.498 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:08:51.669 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:44.499 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:08:51.671 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:44.508 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:08:51.680 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:44.639 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:08:51.817 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:44.641 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:08:51.820 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:44.651 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:08:51.830 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:44.682 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:08:51.861 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:44.684 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:08:51.863 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:44.693 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:08:51.872 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:51.216 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2024-05-15T12:50:51.216 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [----------] Global test environment set-up. 2024-05-15T12:50:51.216 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2024-05-15T12:50:51.216 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2024-05-15T12:50:51.216 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2024-05-15T12:50:51.216 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-05-15T12:50:51.216 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: 2024-05-15T12:50:51.216 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2024-05-15T12:50:51.216 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: seed 40175 2024-05-15T12:50:51.216 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2024-05-15T12:50:51.216 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (340 ms) 2024-05-15T12:50:51.217 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2024-05-15T12:50:51.217 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (35710 ms) 2024-05-15T12:50:51.217 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2024-05-15T12:50:51.217 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (3077 ms) 2024-05-15T12:50:51.217 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2024-05-15T12:50:51.217 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (405 ms) 2024-05-15T12:50:51.217 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2024-05-15T12:50:51.217 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (227 ms) 2024-05-15T12:50:51.217 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2024-05-15T12:50:51.217 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (46 ms) 2024-05-15T12:50:51.217 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2024-05-15T12:50:51.217 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (35 ms) 2024-05-15T12:50:51.217 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2024-05-15T12:50:51.218 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (115 ms) 2024-05-15T12:50:51.218 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2024-05-15T12:50:51.218 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (223 ms) 2024-05-15T12:50:51.218 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2024-05-15T12:50:51.218 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (565 ms) 2024-05-15T12:50:51.218 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2024-05-15T12:50:51.218 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (128 ms) 2024-05-15T12:50:51.218 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2024-05-15T12:50:51.218 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (111 ms) 2024-05-15T12:50:51.218 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2024-05-15T12:50:51.218 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (1100 ms) 2024-05-15T12:50:51.218 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2024-05-15T12:50:51.219 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: osd_max_attr_size = 0 2024-05-15T12:50:51.219 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2024-05-15T12:50:51.219 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (223097 ms) 2024-05-15T12:50:51.219 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2024-05-15T12:50:51.219 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (11684 ms) 2024-05-15T12:50:51.219 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2024-05-15T12:50:51.219 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: waiting for initial deep scrubs... 2024-05-15T12:50:51.219 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: done waiting, doing copies 2024-05-15T12:50:51.219 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: waiting for final deep scrubs... 2024-05-15T12:50:51.219 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: done waiting 2024-05-15T12:50:51.219 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (77477 ms) 2024-05-15T12:50:51.219 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2024-05-15T12:50:51.219 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (104 ms) 2024-05-15T12:50:51.219 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2024-05-15T12:50:51.219 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (56 ms) 2024-05-15T12:50:51.219 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2024-05-15T12:50:51.220 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (5129 ms) 2024-05-15T12:50:51.220 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2024-05-15T12:50:51.220 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2024-05-15T12:50:51.220 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2024-05-15T12:50:51.220 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2024-05-15T12:50:51.220 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2024-05-15T12:50:51.220 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2024-05-15T12:51:03.991 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:09:11.169 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:04.046 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:09:11.224 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:04.089 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:09:11.267 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:04.101 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:09:11.280 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:04.108 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:09:11.287 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:04.130 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:09:11.301 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:04.133 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:09:11.304 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:04.161 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:09:11.340 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:04.172 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:09:11.343 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:21.419 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP (359630 ms total) 2024-05-15T12:51:21.419 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: 2024-05-15T12:51:21.419 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2024-05-15T12:51:21.419 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2024-05-15T12:51:21.419 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (2074 ms) 2024-05-15T12:51:21.419 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (2074 ms total) 2024-05-15T12:51:21.419 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: 2024-05-15T12:51:21.419 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2024-05-15T12:51:21.419 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2024-05-15T12:51:21.419 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (483 ms) 2024-05-15T12:51:21.419 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2024-05-15T12:51:21.420 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (341 ms) 2024-05-15T12:51:21.420 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (824 ms total) 2024-05-15T12:51:21.420 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: 2024-05-15T12:51:21.420 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2024-05-15T12:51:21.420 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2024-05-15T12:51:21.420 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (838 ms) 2024-05-15T12:51:21.420 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2024-05-15T12:51:21.420 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (54 ms) 2024-05-15T12:51:21.420 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (892 ms total) 2024-05-15T12:51:21.420 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: 2024-05-15T12:51:21.420 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2024-05-15T12:51:21.420 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2024-05-15T12:51:21.420 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (628 ms) 2024-05-15T12:51:21.421 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2024-05-15T12:51:21.421 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (36 ms) 2024-05-15T12:51:21.421 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (664 ms total) 2024-05-15T12:51:21.421 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: 2024-05-15T12:51:21.421 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2024-05-15T12:51:21.421 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2024-05-15T12:51:21.421 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1982 ms) 2024-05-15T12:51:21.421 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1982 ms total) 2024-05-15T12:51:21.421 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: 2024-05-15T12:51:21.421 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [----------] Global test environment tear-down 2024-05-15T12:51:21.421 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (395741 ms total) 2024-05-15T12:51:21.421 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ PASSED ] 31 tests. 2024-05-15T12:51:24.241 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:09:31.419 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:24.248 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:09:31.426 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:24.287 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:09:31.466 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:24.292 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:09:31.463 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:24.293 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:09:31.465 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:24.294 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:09:31.473 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:24.299 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:09:31.477 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:24.333 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:09:31.504 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:24.342 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:09:31.521 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:44.331 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:09:51.509 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:44.334 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:09:51.512 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:44.371 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:09:51.549 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:44.386 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:09:51.557 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:44.387 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:09:51.559 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:44.388 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:09:51.567 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:44.394 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:09:51.572 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:44.421 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:09:51.593 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:44.430 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:09:51.609 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:04.416 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:10:11.594 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:04.419 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:10:11.598 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:04.459 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:10:11.637 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:04.465 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:10:11.636 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:04.468 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:10:11.640 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:04.480 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:10:11.659 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:04.483 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:10:11.661 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:04.510 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:10:11.682 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:04.526 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:10:11.705 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:24.581 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:10:31.759 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:24.583 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:10:31.761 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:24.621 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:10:31.799 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:24.636 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:10:31.815 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:24.640 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:10:31.819 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:24.655 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:10:31.827 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:24.667 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:10:31.839 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:24.681 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:10:31.860 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:24.708 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:10:31.880 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:44.534 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:10:51.712 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:44.537 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:10:51.716 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:44.586 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:10:51.757 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:44.586 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:10:51.764 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:44.586 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:10:51.758 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:44.596 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:10:51.775 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:44.601 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:10:51.780 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:44.625 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:10:51.797 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:44.643 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:10:51.821 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:52.400 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:10:59.578 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:52.411 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:10:59.589 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:52.418 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:10:59.590 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:52.420 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:10:59.599 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:52.421 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:10:59.592 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:52.423 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:10:59.602 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:52.507 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:10:59.686 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:52.510 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:10:59.689 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:52.510 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:10:59.682 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:09.627 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:11:16.805 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:09.630 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:11:16.808 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:09.662 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:11:16.840 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:09.677 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:11:16.848 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:09.678 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:11:16.850 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:09.683 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:11:16.862 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:09.685 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:11:16.863 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:09.707 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:11:16.879 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:09.716 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:11:16.895 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:29.664 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:11:36.842 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:29.667 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:11:36.846 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:29.690 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:11:36.868 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:29.715 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:11:36.887 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:29.716 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:11:36.888 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:29.720 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:11:36.898 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:29.725 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:11:36.903 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:29.733 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:11:36.905 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:29.744 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:11:36.922 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:49.760 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:11:56.938 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:49.764 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:11:56.942 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:49.764 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:11:56.936 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:49.770 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:11:56.942 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:49.801 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:11:56.980 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:49.816 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:11:56.994 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:49.817 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:11:56.996 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:49.822 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:11:56.993 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:49.854 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:11:57.032 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:19.908 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:12:27.086 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:19.909 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:12:27.088 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:19.960 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:12:27.131 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:19.962 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:12:27.141 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:19.964 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:12:27.143 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:19.969 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:12:27.147 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:19.979 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:12:27.151 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:20.024 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:12:27.203 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:20.031 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:12:27.203 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:30.103 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:12:37.281 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:30.104 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:12:37.282 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:30.137 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:12:37.316 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:30.163 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:12:37.335 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:30.165 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:12:37.336 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:30.170 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:12:37.349 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:30.177 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:12:37.356 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:30.197 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:12:37.369 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:30.219 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:12:37.397 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:40.218 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:12:47.396 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:40.251 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:12:47.429 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:40.279 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:12:47.458 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:40.296 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:12:47.467 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:40.297 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:12:47.469 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:40.317 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:12:47.496 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:40.320 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:12:47.491 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:40.320 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:12:47.499 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:40.347 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:12:47.525 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:46.758 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2024-05-15T12:54:46.758 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [----------] Global test environment set-up. 2024-05-15T12:54:46.758 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2024-05-15T12:54:46.758 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: seed 40236 2024-05-15T12:54:46.759 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2024-05-15T12:54:46.759 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (35380 ms) 2024-05-15T12:54:46.759 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2024-05-15T12:54:46.759 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (112710 ms) 2024-05-15T12:54:46.759 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2024-05-15T12:54:46.759 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (43 ms) 2024-05-15T12:54:46.759 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (148133 ms total) 2024-05-15T12:54:46.759 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: 2024-05-15T12:54:46.759 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2024-05-15T12:54:46.759 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2024-05-15T12:54:46.759 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (40410 ms) 2024-05-15T12:54:46.759 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2024-05-15T12:54:46.759 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (39000 ms) 2024-05-15T12:54:46.759 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2024-05-15T12:54:46.760 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (25249 ms) 2024-05-15T12:54:46.760 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2024-05-15T12:54:46.760 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: my_snaps [3] 2024-05-15T12:54:46.760 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: my_snaps [4,3] 2024-05-15T12:54:46.760 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: my_snaps [5,4,3] 2024-05-15T12:54:46.760 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: my_snaps [6,5,4,3] 2024-05-15T12:54:46.760 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: promoting some heads 2024-05-15T12:54:46.760 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: promoting from clones for snap 6 2024-05-15T12:54:46.760 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: promoting from clones for snap 5 2024-05-15T12:54:46.760 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: promoting from clones for snap 4 2024-05-15T12:54:46.760 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: promoting from clones for snap 3 2024-05-15T12:54:46.760 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: waiting for scrubs... 2024-05-15T12:54:46.760 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: done waiting 2024-05-15T12:54:46.760 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (117016 ms) 2024-05-15T12:54:46.760 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2024-05-15T12:54:46.761 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (13591 ms) 2024-05-15T12:54:46.761 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2024-05-15T12:54:46.761 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (10139 ms) 2024-05-15T12:54:46.761 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2024-05-15T12:54:46.761 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (9827 ms) 2024-05-15T12:54:46.761 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2024-05-15T12:54:46.761 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (11107 ms) 2024-05-15T12:54:46.761 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2024-05-15T12:54:46.761 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (12930 ms) 2024-05-15T12:54:46.761 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2024-05-15T12:54:46.761 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (11088 ms) 2024-05-15T12:54:46.761 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2024-05-15T12:54:46.761 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (13704 ms) 2024-05-15T12:54:46.761 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2024-05-15T12:54:46.761 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (39259 ms) 2024-05-15T12:54:46.762 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2024-05-15T12:54:46.762 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (12270 ms) 2024-05-15T12:54:46.762 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2024-05-15T12:54:46.762 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (11148 ms) 2024-05-15T12:54:46.762 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2024-05-15T12:54:46.762 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (16346 ms) 2024-05-15T12:54:46.762 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2024-05-15T12:54:46.762 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (11303 ms) 2024-05-15T12:54:46.762 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2024-05-15T12:54:46.762 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (17203 ms) 2024-05-15T12:54:55.245 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:13:02.423 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:55.248 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:13:02.427 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:55.280 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:13:02.458 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:55.300 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:13:02.478 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:55.305 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:13:02.484 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:55.314 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:13:02.485 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:55.316 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:13:02.487 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:55.339 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:13:02.518 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:55.345 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:13:02.516 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:00.172 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:13:07.350 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:00.176 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:13:07.354 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:00.190 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:13:07.369 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:00.221 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:13:07.393 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:00.225 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:13:07.396 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:00.226 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:13:07.403 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:00.228 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:13:07.407 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:00.242 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:13:07.414 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:00.247 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:13:07.426 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:09.523 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2024-05-15T12:55:09.523 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: hmm, no HitSet yet 2024-05-15T12:55:09.523 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: ok, hit_set contains 168:602f83fe:::foo:head 2024-05-15T12:55:09.523 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (9887 ms) 2024-05-15T12:55:09.523 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2024-05-15T12:55:09.523 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg_num = 32 2024-05-15T12:55:09.523 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 0 ls 1715777704,0 2024-05-15T12:55:09.523 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 1 ls 1715777704,0 2024-05-15T12:55:09.523 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 2 ls 1715777704,0 2024-05-15T12:55:09.523 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 3 ls 1715777704,0 2024-05-15T12:55:09.524 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 4 ls 1715777704,0 2024-05-15T12:55:09.524 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 5 ls 1715777704,0 2024-05-15T12:55:09.524 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 6 ls 1715777704,0 2024-05-15T12:55:09.524 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 7 ls 1715777704,0 2024-05-15T12:55:09.524 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 8 ls 1715777704,0 2024-05-15T12:55:09.524 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 9 ls 1715777704,0 2024-05-15T12:55:09.524 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 10 ls 1715777704,0 2024-05-15T12:55:09.524 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 11 ls 1715777704,0 2024-05-15T12:55:09.524 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 12 ls 1715777704,0 2024-05-15T12:55:09.524 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 13 ls 1715777704,0 2024-05-15T12:55:09.524 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 14 ls 1715777704,0 2024-05-15T12:55:09.524 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 15 ls 1715777704,0 2024-05-15T12:55:09.524 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 16 ls 1715777704,0 2024-05-15T12:55:09.524 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 17 ls 1715777704,0 2024-05-15T12:55:09.525 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 18 ls 1715777704,0 2024-05-15T12:55:09.525 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 19 ls 1715777704,0 2024-05-15T12:55:09.525 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 20 ls 1715777704,0 2024-05-15T12:55:09.525 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 21 ls 1715777704,0 2024-05-15T12:55:09.525 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 22 ls 1715777704,0 2024-05-15T12:55:09.525 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 23 ls 1715777704,0 2024-05-15T12:55:09.525 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 24 ls 1715777704,0 2024-05-15T12:55:09.525 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 25 ls 1715777704,0 2024-05-15T12:55:09.525 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 26 ls 1715777704,0 2024-05-15T12:55:09.525 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 27 ls 1715777704,0 2024-05-15T12:55:09.525 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 28 ls 1715777704,0 2024-05-15T12:55:09.525 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 29 ls 1715777704,0 2024-05-15T12:55:09.525 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 30 ls 1715777704,0 2024-05-15T12:55:09.525 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 31 ls 1715777704,0 2024-05-15T12:55:09.525 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg_num = 32 2024-05-15T12:55:09.526 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:6cac518f:::0:head 2024-05-15T12:55:09.526 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:02547ec2:::1:head 2024-05-15T12:55:09.526 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:f905c69b:::2:head 2024-05-15T12:55:09.526 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:cfc208b3:::3:head 2024-05-15T12:55:09.526 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:d83876eb:::4:head 2024-05-15T12:55:09.526 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:b29083e3:::5:head 2024-05-15T12:55:09.526 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:c4fdafeb:::6:head 2024-05-15T12:55:09.526 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:5c6b0b28:::7:head 2024-05-15T12:55:09.526 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:bd63b0f1:::8:head 2024-05-15T12:55:09.526 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:e960b815:::9:head 2024-05-15T12:55:09.526 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:52ea6a34:::10:head 2024-05-15T12:55:09.526 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:89d3ae78:::11:head 2024-05-15T12:55:09.526 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:de5d7c5f:::12:head 2024-05-15T12:55:09.526 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:566253c9:::13:head 2024-05-15T12:55:09.527 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:62a1935d:::14:head 2024-05-15T12:55:09.527 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:863748b0:::15:head 2024-05-15T12:55:09.527 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:3958e169:::16:head 2024-05-15T12:55:09.527 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:4d4dabf9:::17:head 2024-05-15T12:55:09.527 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:8391935d:::18:head 2024-05-15T12:55:09.527 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:28883081:::19:head 2024-05-15T12:55:09.527 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:69259c59:::20:head 2024-05-15T12:55:09.527 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:4bdb80b7:::21:head 2024-05-15T12:55:09.527 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:a11c5d71:::22:head 2024-05-15T12:55:09.527 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:271af37b:::23:head 2024-05-15T12:55:09.527 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:95b121be:::24:head 2024-05-15T12:55:09.527 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:58d1031b:::25:head 2024-05-15T12:55:09.527 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:0a050783:::26:head 2024-05-15T12:55:09.527 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:c709704c:::27:head 2024-05-15T12:55:09.527 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:cbe56eaf:::28:head 2024-05-15T12:55:09.527 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:86b4b162:::29:head 2024-05-15T12:55:09.528 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:70d89383:::30:head 2024-05-15T12:55:09.528 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:dd450c7c:::31:head 2024-05-15T12:55:09.528 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:6d5729b1:::32:head 2024-05-15T12:55:09.528 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:c388f3fb:::33:head 2024-05-15T12:55:09.528 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:56cfea31:::34:head 2024-05-15T12:55:09.528 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:9dbc1bf7:::35:head 2024-05-15T12:55:09.528 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:40b74ccd:::36:head 2024-05-15T12:55:09.528 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:4d5aaf42:::37:head 2024-05-15T12:55:09.528 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:920f362c:::38:head 2024-05-15T12:55:09.528 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:6cc53222:::39:head 2024-05-15T12:55:09.528 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:9cad833f:::40:head 2024-05-15T12:55:09.528 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:1ea84d41:::41:head 2024-05-15T12:55:09.528 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:c4480ef6:::42:head 2024-05-15T12:55:09.528 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:a694361e:::43:head 2024-05-15T12:55:09.528 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:d1bd33e9:::44:head 2024-05-15T12:55:09.529 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:ddc2cd5d:::45:head 2024-05-15T12:55:09.529 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:2b782207:::46:head 2024-05-15T12:55:09.529 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:7b187fca:::47:head 2024-05-15T12:55:09.529 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:90ecdf6f:::48:head 2024-05-15T12:55:09.529 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:a5ed95fe:::49:head 2024-05-15T12:55:09.529 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:ea0eaa55:::50:head 2024-05-15T12:55:09.529 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:f33ef17b:::51:head 2024-05-15T12:55:09.529 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:a0d1b2f6:::52:head 2024-05-15T12:55:09.529 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:60c5229e:::53:head 2024-05-15T12:55:09.529 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:edcbc575:::54:head 2024-05-15T12:55:09.529 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:102cf253:::55:head 2024-05-15T12:55:09.529 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:efb7fb0b:::56:head 2024-05-15T12:55:09.529 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:50d0a326:::57:head 2024-05-15T12:55:09.529 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:d4dc5daf:::58:head 2024-05-15T12:55:09.529 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:3a130462:::59:head 2024-05-15T12:55:09.530 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:ec87ed71:::60:head 2024-05-15T12:55:09.530 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:d5bc9454:::61:head 2024-05-15T12:55:09.530 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:3ddfe313:::62:head 2024-05-15T12:55:09.530 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:7c2816b9:::63:head 2024-05-15T12:55:09.530 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:47e00e4d:::64:head 2024-05-15T12:55:09.530 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:c6410c18:::65:head 2024-05-15T12:55:09.530 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:b48ed237:::66:head 2024-05-15T12:55:09.530 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:cd63ad31:::67:head 2024-05-15T12:55:09.530 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:b179e92b:::68:head 2024-05-15T12:55:09.530 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:0d9f741a:::69:head 2024-05-15T12:55:09.530 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:6d3352ae:::70:head 2024-05-15T12:55:09.530 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:c6d5c19e:::71:head 2024-05-15T12:55:09.530 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:bc4729c3:::72:head 2024-05-15T12:55:09.530 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:77e930b9:::73:head 2024-05-15T12:55:09.531 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:0abeecfd:::74:head 2024-05-15T12:55:09.531 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:b7c37e15:::75:head 2024-05-15T12:55:09.531 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:b6378398:::76:head 2024-05-15T12:55:09.531 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:02bd68de:::77:head 2024-05-15T12:55:09.531 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:cc795d2d:::78:head 2024-05-15T12:55:09.531 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:630d4fea:::79:head 2024-05-15T12:55:09.531 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:e0d29ef5:::80:head 2024-05-15T12:55:09.531 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:fd6f13d2:::81:head 2024-05-15T12:55:09.531 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:606461d5:::82:head 2024-05-15T12:55:09.531 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:eadbdc43:::83:head 2024-05-15T12:55:09.531 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:8761d0bb:::84:head 2024-05-15T12:55:09.531 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:9ef0186f:::85:head 2024-05-15T12:55:09.531 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:e0d41294:::86:head 2024-05-15T12:55:09.531 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:961de695:::87:head 2024-05-15T12:55:09.531 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:1423148f:::88:head 2024-05-15T12:55:09.531 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:633a8fa2:::89:head 2024-05-15T12:55:09.532 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:a8653809:::90:head 2024-05-15T12:55:09.532 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:3dac8b33:::91:head 2024-05-15T12:55:09.532 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:35aad435:::92:head 2024-05-15T12:55:09.532 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:f6dcc343:::93:head 2024-05-15T12:55:09.532 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:dbbdad87:::94:head 2024-05-15T12:55:09.532 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:1cb48ce0:::95:head 2024-05-15T12:55:09.532 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:03cd461c:::96:head 2024-05-15T12:55:09.532 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:17a4ea99:::97:head 2024-05-15T12:55:09.532 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:9993c9a7:::98:head 2024-05-15T12:55:09.532 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:6394211c:::99:head 2024-05-15T12:55:09.532 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:94c7ae57:::100:head 2024-05-15T12:55:09.532 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:6fdee5bb:::101:head 2024-05-15T12:55:09.532 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:9a477fd1:::102:head 2024-05-15T12:55:09.532 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:eb850916:::103:head 2024-05-15T12:55:09.532 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:affc56b9:::104:head 2024-05-15T12:55:09.532 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:b42dc814:::105:head 2024-05-15T12:55:09.533 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:f319f8f0:::106:head 2024-05-15T12:55:09.533 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:9a40b9de:::107:head 2024-05-15T12:55:09.533 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:8b524f28:::108:head 2024-05-15T12:55:09.533 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:e3de589f:::109:head 2024-05-15T12:55:09.533 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:90f90a5b:::110:head 2024-05-15T12:55:09.533 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:a7b4f1d7:::111:head 2024-05-15T12:55:09.533 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:af51766e:::112:head 2024-05-15T12:55:09.533 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:b6f90bd1:::113:head 2024-05-15T12:55:09.533 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:e0261208:::114:head 2024-05-15T12:55:09.533 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:c9569ef7:::115:head 2024-05-15T12:55:09.533 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:61bebe50:::116:head 2024-05-15T12:55:09.533 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:fe93412b:::117:head 2024-05-15T12:55:09.533 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:d3d38bee:::118:head 2024-05-15T12:55:09.533 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:3100ba0c:::119:head 2024-05-15T12:55:09.533 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:d0560ada:::120:head 2024-05-15T12:55:09.534 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:f0ea8b35:::121:head 2024-05-15T12:55:09.534 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:766f231a:::122:head 2024-05-15T12:55:09.534 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:a07a2582:::123:head 2024-05-15T12:55:09.534 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:bd7c6b3a:::124:head 2024-05-15T12:55:09.534 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:fb2ddaff:::125:head 2024-05-15T12:55:09.534 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:4408e1fe:::126:head 2024-05-15T12:55:09.534 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:ee1df7a7:::127:head 2024-05-15T12:55:09.534 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:c3002909:::128:head 2024-05-15T12:55:09.534 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:4f48ffa9:::129:head 2024-05-15T12:55:09.534 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:edf38733:::130:head 2024-05-15T12:55:09.534 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:c08425c0:::131:head 2024-05-15T12:55:09.534 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:5f902d98:::132:head 2024-05-15T12:55:09.534 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:41ea2c93:::133:head 2024-05-15T12:55:09.534 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:813cee13:::134:head 2024-05-15T12:55:09.534 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:0131818d:::135:head 2024-05-15T12:55:09.535 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:26ba5a85:::136:head 2024-05-15T12:55:09.535 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:381b8a5a:::137:head 2024-05-15T12:55:09.535 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:28797e47:::138:head 2024-05-15T12:55:09.535 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:bfca7f22:::139:head 2024-05-15T12:55:09.535 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:36807075:::140:head 2024-05-15T12:55:09.535 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:80b03975:::141:head 2024-05-15T12:55:09.535 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:5c15709b:::142:head 2024-05-15T12:55:09.535 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:f39ea15e:::143:head 2024-05-15T12:55:09.535 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:ea992956:::144:head 2024-05-15T12:55:09.535 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:48887b1c:::145:head 2024-05-15T12:55:09.535 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:9f24a9dd:::146:head 2024-05-15T12:55:09.535 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:987f100b:::147:head 2024-05-15T12:55:09.535 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:d2dd3581:::148:head 2024-05-15T12:55:09.535 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:7fed1808:::149:head 2024-05-15T12:55:09.535 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:c80b70e9:::150:head 2024-05-15T12:55:09.536 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:85ed90f9:::151:head 2024-05-15T12:55:09.536 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:36428b24:::152:head 2024-05-15T12:55:09.536 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:d044c34a:::153:head 2024-05-15T12:55:09.536 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:7c18bf58:::154:head 2024-05-15T12:55:09.536 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:d1c21232:::155:head 2024-05-15T12:55:09.536 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:a7a3c575:::156:head 2024-05-15T12:55:09.536 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:87da0633:::157:head 2024-05-15T12:55:09.536 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:d5ac3822:::158:head 2024-05-15T12:55:09.536 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:3f20522d:::159:head 2024-05-15T12:55:09.536 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:6ca26563:::160:head 2024-05-15T12:55:09.536 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:532ce135:::161:head 2024-05-15T12:55:09.536 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:c78863e6:::162:head 2024-05-15T12:55:10.156 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:13:17.334 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:10.243 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:13:17.422 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:10.260 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:13:17.439 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:10.263 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:13:17.441 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:10.263 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:13:17.442 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:10.266 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:13:17.437 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:10.269 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:13:17.440 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:10.287 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:13:17.466 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:10.290 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:13:17.462 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:14.721 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:13:21.899 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:14.725 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:13:21.904 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:14.742 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:13:21.921 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:14.783 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:13:21.955 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:14.784 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:13:21.962 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:14.784 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:13:21.956 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:14.785 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:13:21.964 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:14.801 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:13:21.972 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:14.802 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:13:21.981 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:20.463 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:13:27.642 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:20.465 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:13:27.643 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:20.482 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:13:27.660 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:20.514 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:13:27.693 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:20.521 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:13:27.699 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:20.526 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:13:27.698 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:20.531 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:13:27.703 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:20.541 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:13:27.720 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:20.550 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:13:27.721 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:25.288 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:13:32.466 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:25.292 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:13:32.471 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:25.308 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:13:32.487 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:25.336 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:13:32.508 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:25.338 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:13:32.509 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:25.340 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:13:32.519 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:25.342 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:13:32.521 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:25.353 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:13:32.524 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:25.359 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:13:32.537 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:40.321 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:13:47.499 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:40.326 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:13:47.505 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:40.328 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:13:47.499 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:40.331 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:13:47.503 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:40.351 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:13:47.523 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:40.361 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:13:47.539 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:40.378 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:13:47.557 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:40.383 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:13:47.561 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:40.404 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:13:47.582 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:45.400 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:13:52.578 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:45.407 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:13:52.585 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:45.426 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:13:52.604 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:45.477 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:13:52.649 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:45.479 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:13:52.651 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:45.480 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:13:52.658 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:45.482 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:13:52.660 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:45.501 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:13:52.672 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:45.506 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:13:52.685 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:00.434 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:14:07.612 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:00.441 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:14:07.620 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:00.492 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:14:07.671 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:00.505 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:14:07.676 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:00.506 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:14:07.678 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:00.520 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:14:07.698 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:00.524 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:14:07.702 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:00.542 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:14:07.714 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:00.564 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:14:07.743 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:01.099 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:14:08.270 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:01.100 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:14:08.271 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:01.107 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:14:08.279 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:01.169 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:14:08.347 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:01.170 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:14:08.349 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:01.180 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:14:08.359 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:01.304 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:14:08.482 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:01.305 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:14:08.484 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:01.315 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:14:08.494 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:15.492 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:14:22.670 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:15.493 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:14:22.672 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:15.522 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:14:22.700 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:15.543 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:14:22.714 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:15.544 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:14:22.716 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:15.546 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:14:22.724 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:15.548 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:14:22.726 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:15.571 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:14:22.743 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:15.574 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:14:22.753 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:20.698 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:14:27.877 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:20.702 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:14:27.880 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:20.718 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:14:27.896 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:20.769 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:14:27.940 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:20.770 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:14:27.942 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:20.773 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:14:27.952 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:20.780 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:14:27.959 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:20.782 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:14:27.954 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:20.795 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:14:27.973 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:22.687 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:7462ddf6:::163:head 2024-05-15T12:56:22.688 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:5d165639:::164:head 2024-05-15T12:56:22.688 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:f43765fc:::165:head 2024-05-15T12:56:22.688 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:b4c720e9:::166:head 2024-05-15T12:56:22.688 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:e694b040:::167:head 2024-05-15T12:56:22.688 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:afa38db2:::168:head 2024-05-15T12:56:22.688 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:77ba9f53:::169:head 2024-05-15T12:56:22.688 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:87495034:::170:head 2024-05-15T12:56:22.688 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:7c96bf0e:::171:head 2024-05-15T12:56:22.688 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:dbe346cc:::172:head 2024-05-15T12:56:22.688 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:e943ec24:::173:head 2024-05-15T12:56:22.688 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:f97a9c0c:::174:head 2024-05-15T12:56:22.688 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:6f26e74d:::175:head 2024-05-15T12:56:22.689 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:4f95e106:::176:head 2024-05-15T12:56:22.689 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:0e6f2f8f:::177:head 2024-05-15T12:56:22.689 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:05db05f1:::178:head 2024-05-15T12:56:22.689 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:38a78d66:::179:head 2024-05-15T12:56:22.689 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:d095610b:::180:head 2024-05-15T12:56:22.689 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:a1a9d709:::181:head 2024-05-15T12:56:22.689 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:1e5d39db:::182:head 2024-05-15T12:56:22.689 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:f7df4fb9:::183:head 2024-05-15T12:56:22.689 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:03a7f161:::184:head 2024-05-15T12:56:22.689 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:ba70721e:::185:head 2024-05-15T12:56:22.689 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:28e5662d:::186:head 2024-05-15T12:56:22.689 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:973d52de:::187:head 2024-05-15T12:56:22.689 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:4303eb1c:::188:head 2024-05-15T12:56:22.689 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:b990b48e:::189:head 2024-05-15T12:56:22.690 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:29b8165b:::190:head 2024-05-15T12:56:22.690 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:3547f197:::191:head 2024-05-15T12:56:22.690 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:7e260936:::192:head 2024-05-15T12:56:22.690 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:1abec7b1:::193:head 2024-05-15T12:56:22.690 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:10fdda93:::194:head 2024-05-15T12:56:22.690 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:15817eea:::195:head 2024-05-15T12:56:22.690 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:770bab57:::196:head 2024-05-15T12:56:22.690 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:ed9e13e7:::197:head 2024-05-15T12:56:22.690 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:71471a8f:::198:head 2024-05-15T12:56:22.690 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 169:10fb1d02:::199:head 2024-05-15T12:56:22.690 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (22804 ms) 2024-05-15T12:56:22.690 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2024-05-15T12:56:22.690 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: got ls 1715777727,0 2024-05-15T12:56:22.690 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: first is 1715777727 2024-05-15T12:56:22.691 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: got ls 1715777727,0 2024-05-15T12:56:22.691 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: got ls 1715777727,0 2024-05-15T12:56:22.691 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: got ls 1715777727,1715777730,1715777731,0 2024-05-15T12:56:22.691 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: got ls 1715777727,1715777730,1715777731,0 2024-05-15T12:56:22.691 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: got ls 1715777727,1715777730,1715777731,0 2024-05-15T12:56:22.691 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: got ls 1715777727,1715777730,1715777731,1715777733,1715777734,0 2024-05-15T12:56:22.691 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: got ls 1715777727,1715777730,1715777731,1715777733,1715777734,0 2024-05-15T12:56:22.691 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: got ls 1715777727,1715777730,1715777731,1715777733,1715777734,0 2024-05-15T12:56:22.691 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: got ls 1715777727,1715777730,1715777731,1715777733,1715777734,1715777736,1715777737,0 2024-05-15T12:56:22.691 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: got ls 1715777727,1715777730,1715777731,1715777733,1715777734,1715777736,1715777737,0 2024-05-15T12:56:22.691 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: got ls 1715777727,1715777730,1715777731,1715777733,1715777734,1715777736,1715777737,0 2024-05-15T12:56:22.691 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: got ls 1715777731,1715777733,1715777734,1715777736,1715777737,1715777739,1715777740,0 2024-05-15T12:56:22.691 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: first now 1715777731, trimmed 2024-05-15T12:56:22.691 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (23541 ms) 2024-05-15T12:56:22.691 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2024-05-15T12:56:22.691 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: foo0 2024-05-15T12:56:22.692 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-05-15T12:56:22.692 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (19546 ms) 2024-05-15T12:56:22.692 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2024-05-15T12:56:30.658 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:14:37.836 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:30.660 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:14:37.838 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:30.684 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:14:37.863 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:30.714 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:14:37.886 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:30.717 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:14:37.888 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:30.720 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:14:37.898 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:30.721 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:14:37.900 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:30.744 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:14:37.915 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:30.747 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:14:37.926 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:40.672 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:14:47.851 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:40.674 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:14:47.853 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:40.701 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:14:47.880 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:40.723 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:14:47.895 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:40.724 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:14:47.902 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:40.724 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:14:47.896 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:40.725 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:14:47.903 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:40.748 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:14:47.920 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:40.749 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:14:47.927 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:45.832 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:14:53.011 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:45.860 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:14:53.038 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:45.882 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:14:53.061 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:45.885 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:14:53.056 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:45.887 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:14:53.059 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:45.900 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:14:53.071 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:45.912 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:14:53.090 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:45.919 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:14:53.098 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:45.941 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:14:53.120 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:05.801 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:15:12.979 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:05.871 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:15:13.043 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:05.874 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:15:13.046 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:05.892 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:15:13.071 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:05.932 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:15:13.104 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:05.940 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:15:13.118 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:05.948 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:15:13.127 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:05.973 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:15:13.151 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:06.007 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:15:13.185 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:16.120 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:15:23.298 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:16.128 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:15:23.307 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:16.130 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:15:23.308 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:16.164 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:15:23.343 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:16.193 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:15:23.365 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:16.193 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:15:23.372 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:16.199 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:15:23.370 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:16.232 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:15:23.403 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:16.370 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:15:23.548 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:26.146 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:15:33.325 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:26.149 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:15:33.327 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:26.179 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:15:33.357 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:26.200 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:15:33.371 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:26.204 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:15:33.376 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:26.218 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:15:33.397 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:26.220 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:15:33.399 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:26.226 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:15:33.397 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:26.252 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:15:33.430 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:31.114 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:15:38.293 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:31.116 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:15:38.294 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:31.131 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:15:38.310 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:31.166 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:15:38.338 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:31.169 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:15:38.341 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:31.171 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:15:38.350 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:31.172 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:15:38.351 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:31.181 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:15:38.353 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:31.184 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:15:38.363 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:46.045 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:15:53.223 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:46.047 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:15:53.226 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:46.072 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:15:53.250 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:46.098 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:15:53.270 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:46.101 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:15:53.273 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:46.108 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:15:53.287 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:46.111 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:15:53.290 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:46.128 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:15:53.300 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:46.141 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:15:53.319 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:46.699 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:15:53.871 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:46.700 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:15:53.872 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:46.709 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:15:53.880 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:46.731 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:15:53.910 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:46.733 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:15:53.911 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:46.740 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:15:53.919 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:46.755 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:15:53.933 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:46.756 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:15:53.935 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:46.765 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:15:53.944 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:01.063 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:16:08.241 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:01.124 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:16:08.303 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:01.170 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:16:08.349 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:01.186 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:16:08.357 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:01.187 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:16:08.359 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:01.187 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:16:08.366 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:01.198 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:16:08.377 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:01.218 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:16:08.390 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:01.248 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:16:08.427 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:06.081 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:16:13.259 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:06.133 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:16:13.311 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:06.145 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:16:13.324 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:06.177 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:16:13.349 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:06.178 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:16:13.350 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:06.189 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:16:13.361 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:06.191 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:16:13.370 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:06.195 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:16:13.374 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:06.206 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:16:13.385 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:21.446 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:16:28.624 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:21.452 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:16:28.631 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:21.483 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:16:28.662 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:21.525 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:16:28.703 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:21.526 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:16:28.698 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:21.527 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:16:28.705 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:21.531 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:16:28.702 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:21.564 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:16:28.742 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:21.585 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:16:28.757 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:26.307 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:16:33.486 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:26.363 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:16:33.535 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:26.367 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:16:33.539 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:26.369 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:16:33.547 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:26.384 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:16:33.555 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:26.398 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:16:33.577 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:26.456 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:16:33.635 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:26.460 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:16:33.638 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:26.495 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:16:33.673 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:36.273 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:16:43.451 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:36.308 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:16:43.487 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:36.322 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:16:43.494 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:36.324 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:16:43.496 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:36.327 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:16:43.506 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:36.328 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:16:43.507 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:36.330 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:16:43.509 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:36.341 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:16:43.512 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:36.345 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:16:43.524 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:46.301 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:16:53.479 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:46.339 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:16:53.517 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:46.353 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:16:53.525 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:46.354 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:16:53.532 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:46.354 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:16:53.533 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:46.355 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:16:53.527 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:46.356 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:16:53.535 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:46.369 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:16:53.540 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:46.370 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:16:53.548 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:56.522 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:17:03.700 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:56.526 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:17:03.705 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:56.548 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:17:03.727 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:56.575 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:17:03.747 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:56.576 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:17:03.748 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:56.582 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:17:03.760 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:56.583 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:17:03.762 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:56.599 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:17:03.771 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:56.619 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:17:03.797 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:06.366 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:17:13.545 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:06.370 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:17:13.548 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:06.393 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:17:13.572 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:06.417 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:17:13.588 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:06.418 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:17:13.590 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:06.420 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:17:13.599 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:06.424 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:17:13.603 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:06.434 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:17:13.606 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:06.442 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:17:13.621 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:16.459 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:17:23.638 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:16.464 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:17:23.643 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:16.501 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:17:23.679 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:16.537 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:17:23.708 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:16.538 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:17:23.710 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:16.540 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:17:23.719 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:16.543 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:17:23.722 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:16.568 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:17:23.740 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:16.575 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:17:23.753 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:17.196 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:17:24.374 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:17.198 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:17:24.375 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:17.203 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:17:24.375 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:17.205 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:17:24.377 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:17.216 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:17:24.388 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:17.234 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:17:24.412 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:17.431 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:17:24.610 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:17.433 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:17:24.612 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:17.446 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:17:24.625 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:26.487 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:17:33.665 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:26.488 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:17:33.667 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:26.527 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:17:33.706 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:26.538 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:17:33.710 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:26.539 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:17:33.711 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:26.554 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:17:33.733 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:26.555 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:17:33.734 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:26.566 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:17:33.737 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:26.580 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:17:33.759 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:31.614 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:17:38.792 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:31.618 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:17:38.797 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:31.639 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:17:38.818 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:31.672 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:17:38.844 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:31.674 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:17:38.846 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:31.680 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:17:38.859 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:31.681 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:17:38.860 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:31.691 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:17:38.862 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:31.704 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:17:38.882 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:46.550 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:17:53.728 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:46.552 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:17:53.731 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:46.570 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:17:53.749 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:46.604 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:17:53.782 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:46.604 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:17:53.775 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:46.605 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:17:53.777 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:46.607 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:17:53.786 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:46.625 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:17:53.803 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:46.625 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:17:53.797 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:48.036 INFO:tasks.ceph.osd.3.smithi117.stderr:2024-05-15T12:59:48.027+0000 2268b640 -1 snap_mapper.add_oid found existing snaps mapped on 187:01a3fcc7:test-rados-api-smithi117-40236-80::foo:2, removing 2024-05-15T12:59:48.039 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-05-15T12:59:48.042+0000 2ca9b640 -1 snap_mapper.add_oid found existing snaps mapped on 187:01a3fcc7:test-rados-api-smithi117-40236-80::foo:2, removing 2024-05-15T12:59:51.558 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:17:58.736 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:51.560 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:17:58.739 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:51.574 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:17:58.752 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:51.607 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:17:58.779 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:51.608 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:17:58.780 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:51.612 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:17:58.790 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:51.616 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:17:58.794 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:51.622 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:17:58.794 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:51.629 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:17:58.808 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:06.645 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:18:13.823 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:06.647 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:18:13.825 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:06.680 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:18:13.858 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:06.693 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:18:13.864 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:06.695 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:18:13.867 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:06.697 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:18:13.876 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:06.698 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:18:13.877 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:06.731 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:18:13.903 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:06.734 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:18:13.913 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:10.646 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:18:17.824 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:10.654 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:18:17.832 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:10.663 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:18:17.842 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:10.664 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:18:17.836 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:10.710 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:18:17.881 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:10.730 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:18:17.902 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:10.738 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:18:17.916 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:10.758 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:18:17.937 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:10.768 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:18:17.947 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:11.615 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:18:18.794 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:11.621 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:18:18.792 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:11.622 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:18:18.801 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:11.623 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:18:18.801 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:11.626 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:18:18.798 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:11.670 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:18:18.849 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:11.687 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:18:18.866 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:11.695 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:18:18.874 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:11.706 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:18:18.878 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:21.806 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:18:28.984 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:21.807 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:18:28.986 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:21.832 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:18:29.011 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:21.856 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:18:29.028 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:21.857 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:18:29.029 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:21.868 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:18:29.047 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:21.871 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:18:29.050 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:21.881 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:18:29.053 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:21.905 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:18:29.084 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:25.826 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ProxyRead (20152 ms) 2024-05-15T13:00:25.826 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2024-05-15T13:00:25.826 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (25235 ms) 2024-05-15T13:00:25.826 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2024-05-15T13:00:25.826 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (5502 ms) 2024-05-15T13:00:25.826 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2024-05-15T13:00:25.826 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: require_osd_release = squid 2024-05-15T13:00:25.826 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (5588 ms) 2024-05-15T13:00:25.827 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2024-05-15T13:00:25.827 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (4408 ms) 2024-05-15T13:00:25.827 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2024-05-15T13:00:25.827 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: require_osd_release = squid 2024-05-15T13:00:25.827 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (5785 ms) 2024-05-15T13:00:25.827 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2024-05-15T13:00:25.827 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: require_osd_release = squid 2024-05-15T13:00:25.827 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (4914 ms) 2024-05-15T13:00:25.827 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2024-05-15T13:00:25.827 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: require_osd_release = squid 2024-05-15T13:00:25.827 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (40238 ms) 2024-05-15T13:00:25.827 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2024-05-15T13:00:25.827 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: require_osd_release = squid 2024-05-15T13:00:25.827 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (18275 ms) 2024-05-15T13:00:25.827 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2024-05-15T13:00:25.828 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: require_osd_release = squid 2024-05-15T13:00:25.828 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (5888 ms) 2024-05-15T13:00:25.828 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2024-05-15T13:00:25.828 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: require_osd_release = squid 2024-05-15T13:00:25.828 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (5324 ms) 2024-05-15T13:00:25.828 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2024-05-15T13:00:25.828 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: require_osd_release = squid 2024-05-15T13:00:25.828 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (27695 ms) 2024-05-15T13:00:25.828 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2024-05-15T13:00:25.828 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: require_osd_release = squid 2024-05-15T13:00:25.828 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (18032 ms) 2024-05-15T13:00:25.828 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2024-05-15T13:00:25.828 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: require_osd_release = squid 2024-05-15T13:00:25.829 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (6631 ms) 2024-05-15T13:00:25.829 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2024-05-15T13:00:25.829 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: require_osd_release = squid 2024-05-15T13:00:25.829 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (6328 ms) 2024-05-15T13:00:25.829 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2024-05-15T13:00:25.829 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: require_osd_release = squid 2024-05-15T13:00:25.829 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: waiting for scrubs... 2024-05-15T13:00:25.829 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: done waiting 2024-05-15T13:00:25.829 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (27106 ms) 2024-05-15T13:00:25.829 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2024-05-15T13:00:25.829 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: require_osd_release = squid 2024-05-15T13:00:25.829 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (13049 ms) 2024-05-15T13:00:25.829 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2024-05-15T13:00:25.829 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: require_osd_release = squid 2024-05-15T13:00:25.829 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (11533 ms) 2024-05-15T13:00:25.829 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2024-05-15T13:00:25.830 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: require_osd_release = squid 2024-05-15T13:00:25.830 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (11608 ms) 2024-05-15T13:00:31.854 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:18:39.032 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:31.858 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:18:39.037 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:31.882 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:18:39.061 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:31.906 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:18:39.078 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:31.907 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:18:39.079 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:31.909 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:18:39.088 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:31.911 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:18:39.090 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:31.932 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:18:39.104 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:31.936 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:18:39.115 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:37.004 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:18:44.182 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:37.007 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:18:44.186 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:37.010 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:18:44.182 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:37.012 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:18:44.184 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:37.013 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:18:44.192 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:37.015 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:18:44.193 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:37.079 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:18:44.258 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:37.081 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:18:44.260 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:37.108 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:18:44.280 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:46.962 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:18:54.141 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:47.031 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:18:54.209 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:47.038 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:18:54.210 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:47.039 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:18:54.211 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:47.063 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:18:54.242 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:47.065 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:18:54.244 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:47.067 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:18:54.246 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:47.070 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:18:54.242 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:47.111 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:18:54.290 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:51.938 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:18:59.114 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:51.940 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:18:59.119 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:51.956 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:18:59.135 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:51.989 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:18:59.168 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:51.990 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:18:59.169 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:51.994 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:18:59.166 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:51.996 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:18:59.168 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:52.006 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:18:59.184 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:52.012 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:18:59.184 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:01.989 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:19:09.167 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:01.990 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:19:09.169 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:02.023 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:19:09.202 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:02.036 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:19:09.207 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:02.037 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:19:09.209 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:02.056 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:19:09.235 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:02.058 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:19:09.237 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:02.058 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:19:09.230 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:02.080 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:19:09.259 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:12.118 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:19:19.297 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:12.165 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:19:19.337 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:12.166 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:19:19.338 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:12.177 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:19:19.356 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:12.178 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:19:19.357 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:12.183 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:19:19.355 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:12.196 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:19:19.375 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:12.202 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:19:19.381 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:12.244 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:19:19.423 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:22.150 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:19:29.329 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:22.154 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:19:29.333 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:22.180 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:19:29.359 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:22.209 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:19:29.379 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:22.213 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:19:29.385 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:22.213 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:19:29.392 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:22.215 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:19:29.394 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:22.235 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:19:29.407 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:22.236 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:19:29.415 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:23.077 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:19:30.248 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:23.078 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:19:30.250 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:23.090 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:19:30.262 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:23.120 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:19:30.299 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:23.121 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:19:30.300 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:23.134 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:19:30.313 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:23.214 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:19:30.393 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:23.216 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:19:30.395 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:23.225 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:19:30.404 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:32.133 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:19:39.311 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:32.182 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:19:39.354 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:32.183 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:19:39.355 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:32.201 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:19:39.380 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:32.201 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:19:39.373 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:32.202 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:19:39.381 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:32.205 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:19:39.384 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:32.228 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:19:39.407 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:32.231 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:19:39.410 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:35.818 INFO:tasks.ceph.osd.1.smithi117.stderr:2024-05-15T13:01:35.798+0000 2628e640 -1 snap_mapper.add_oid found existing snaps mapped on 105:d178b9ca:test-rados-api-smithi117-40236-97::foo:21, removing 2024-05-15T13:01:35.826 INFO:tasks.ceph.osd.5.smithi178.stderr:2024-05-15T13:01:35.829+0000 2a296640 -1 snap_mapper.add_oid found existing snaps mapped on 105:d178b9ca:test-rados-api-smithi117-40236-97::foo:21, removing 2024-05-15T13:01:37.147 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:19:44.326 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:37.197 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:19:44.376 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:37.208 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:19:44.387 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:37.244 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:19:44.416 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:37.245 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:19:44.417 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:37.247 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:19:44.426 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:37.249 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:19:44.428 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:37.256 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:19:44.428 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:37.260 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:19:44.439 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:52.514 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:19:59.693 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:52.578 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:19:59.757 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:52.589 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:19:59.761 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:52.592 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:19:59.764 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:52.595 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:19:59.774 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:52.615 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:19:59.794 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:52.625 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:19:59.804 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:52.643 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:19:59.815 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:52.668 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:19:59.846 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:07.471 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:20:14.650 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:07.523 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:20:14.695 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:07.524 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:20:14.696 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:07.542 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:20:14.721 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:07.562 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:20:14.734 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:07.585 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:20:14.764 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:07.626 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:20:14.805 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:07.630 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:20:14.809 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:07.669 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:20:14.848 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:12.524 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:20:19.703 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:12.530 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:20:19.710 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:12.559 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:20:19.738 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:12.588 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:20:19.759 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:12.589 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:20:19.761 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:12.591 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:20:19.770 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:12.593 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:20:19.772 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:12.607 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:20:19.779 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:12.610 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:20:19.789 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:24.540 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:20:31.719 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:24.545 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:20:31.724 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:24.594 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:20:31.772 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:24.596 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:20:31.768 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:24.597 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:20:31.770 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:24.611 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:20:31.790 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:24.612 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:20:31.792 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:24.630 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:20:31.802 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:24.659 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:20:31.838 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:27.617 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:20:34.795 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:27.620 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:20:34.799 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:27.633 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:20:34.805 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:27.637 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:20:34.809 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:27.653 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:20:34.832 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:27.663 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:20:34.835 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:27.675 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:20:34.854 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:27.678 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:20:34.857 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:27.695 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:20:34.874 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:32.487 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:20:39.665 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:32.536 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:20:39.715 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:32.551 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:20:39.730 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:32.583 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:20:39.755 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:32.584 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:20:39.756 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:32.599 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:20:39.771 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:32.607 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:20:39.786 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:32.608 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:20:39.787 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:32.623 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:20:39.802 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:42.723 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:20:49.901 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:42.727 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:20:49.906 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:42.760 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:20:49.939 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:42.771 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:20:49.943 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:42.772 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:20:49.944 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:42.798 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:20:49.971 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:42.799 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:20:49.978 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:42.801 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:20:49.980 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:42.834 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:20:50.013 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:52.893 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:21:00.072 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:52.951 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:21:00.123 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:52.952 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:21:00.125 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:52.966 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:21:00.145 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:52.996 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:21:00.168 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:53.017 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:21:00.196 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:53.034 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:21:00.214 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:53.048 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:21:00.227 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:53.085 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:21:00.264 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:57.676 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:21:04.855 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:57.679 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:21:04.858 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:57.695 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:21:04.874 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:57.727 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:21:04.899 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:57.729 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:21:04.901 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:57.736 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:21:04.915 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:57.738 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:21:04.917 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:57.746 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:21:04.918 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:57.757 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:21:04.936 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:12.862 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:21:20.041 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:12.874 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:21:20.053 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:12.875 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:21:20.047 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:12.877 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:21:20.056 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:12.878 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:21:20.057 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:12.930 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:21:20.101 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:12.970 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:21:20.142 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:12.977 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:21:20.156 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:13.019 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:21:20.198 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:27.800 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:21:34.978 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:27.804 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:21:34.983 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:27.831 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:21:35.010 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:27.856 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:21:35.028 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:27.858 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:21:35.030 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:27.858 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:21:35.037 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:27.859 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:21:35.038 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:27.881 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:21:35.053 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:27.884 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:21:35.063 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:32.816 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:21:39.995 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:32.818 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:21:39.997 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:32.830 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:21:40.009 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:32.866 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:21:40.038 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:32.868 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:21:40.040 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:32.868 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:21:40.047 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:32.870 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:21:40.049 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:32.879 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:21:40.051 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:32.881 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:21:40.060 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:43.002 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:21:50.181 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:43.006 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:21:50.185 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:43.039 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:21:50.218 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:43.068 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:21:50.240 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:43.072 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:21:50.244 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:43.078 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:21:50.257 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:43.083 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:21:50.262 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:43.101 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:21:50.273 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:43.109 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:21:50.288 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:58.085 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:22:05.263 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:58.087 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:22:05.266 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:58.121 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:22:05.300 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:58.142 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:22:05.321 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:58.143 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:22:05.323 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:58.155 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:22:05.328 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:58.176 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:22:05.347 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:58.190 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:22:05.367 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:58.210 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:22:05.382 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:13.005 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:22:20.184 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:13.010 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:22:20.189 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:13.045 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:22:20.224 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:13.060 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:22:20.232 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:13.061 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:22:20.240 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:13.062 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:22:20.234 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:13.063 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:22:20.243 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:13.093 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:22:20.265 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:13.097 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:22:20.276 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:18.120 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:22:25.299 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:18.177 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:22:25.356 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:18.181 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:22:25.353 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:18.186 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:22:25.358 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:18.208 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:22:25.387 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:18.214 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:22:25.386 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:18.253 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:22:25.432 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:18.255 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:22:25.434 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:18.272 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:22:25.451 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:28.110 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:22:35.289 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:28.114 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:22:35.293 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:28.154 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:22:35.333 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:28.174 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:22:35.346 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:28.182 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:22:35.361 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:28.184 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:22:35.363 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:28.198 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:22:35.370 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:28.219 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:22:35.398 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:28.234 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:22:35.406 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:30.141 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:22:37.320 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:30.197 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:22:37.369 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:30.200 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:22:37.379 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:30.202 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:22:37.374 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:30.218 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:22:37.397 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:30.223 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:22:37.395 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:30.262 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:22:37.441 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:30.264 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:22:37.443 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:30.286 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:22:37.465 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:36.604 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2024-05-15T13:04:36.604 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: require_osd_release = squid 2024-05-15T13:04:36.604 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (12130 ms) 2024-05-15T13:04:36.604 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2024-05-15T13:04:36.604 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: require_osd_release = squid 2024-05-15T13:04:36.604 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (8808 ms) 2024-05-15T13:04:36.605 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2024-05-15T13:04:36.605 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: require_osd_release = squid 2024-05-15T13:04:36.605 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (7278 ms) 2024-05-15T13:04:36.605 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2024-05-15T13:04:36.605 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: require_osd_release = squid 2024-05-15T13:04:36.605 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (27532 ms) 2024-05-15T13:04:36.605 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2024-05-15T13:04:36.605 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: require_osd_release = squid 2024-05-15T13:04:36.605 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (16250 ms) 2024-05-15T13:04:36.605 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2024-05-15T13:04:36.605 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (12098 ms) 2024-05-15T13:04:36.605 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2024-05-15T13:04:36.605 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2024-05-15T13:04:36.605 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: 00000010 69 73 |is| 2024-05-15T13:04:36.605 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: 00000012 2024-05-15T13:04:36.606 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (12698 ms) 2024-05-15T13:04:36.606 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2024-05-15T13:04:36.606 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: require_osd_release = squid 2024-05-15T13:04:36.606 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (6521 ms) 2024-05-15T13:04:36.606 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (853976 ms total) 2024-05-15T13:04:36.606 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: 2024-05-15T13:04:36.606 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2024-05-15T13:04:36.606 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2024-05-15T13:04:36.606 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1874 ms) 2024-05-15T13:04:36.606 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2024-05-15T13:04:36.606 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (12393 ms) 2024-05-15T13:04:36.606 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2024-05-15T13:04:36.606 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (19270 ms) 2024-05-15T13:04:36.606 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2024-05-15T13:04:36.607 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (23 ms) 2024-05-15T13:04:36.607 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (33560 ms total) 2024-05-15T13:04:36.607 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: 2024-05-15T13:04:36.607 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2024-05-15T13:04:36.607 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2024-05-15T13:04:36.607 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (8965 ms) 2024-05-15T13:04:36.607 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2024-05-15T13:04:36.607 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (9916 ms) 2024-05-15T13:04:36.607 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2024-05-15T13:04:36.607 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: waiting for scrub... 2024-05-15T13:04:36.607 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: done waiting 2024-05-15T13:04:36.607 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (28025 ms) 2024-05-15T13:04:36.607 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2024-05-15T13:04:36.607 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (12160 ms) 2024-05-15T13:04:36.607 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2024-05-15T13:04:36.608 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (10487 ms) 2024-05-15T13:04:36.608 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2024-05-15T13:04:36.608 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (11349 ms) 2024-05-15T13:04:36.608 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2024-05-15T13:04:36.608 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (12789 ms) 2024-05-15T13:04:36.608 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2024-05-15T13:04:38.385 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:22:45.564 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:38.422 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:22:45.601 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:38.445 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:22:45.624 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:38.446 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:22:45.625 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:38.455 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:22:45.634 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:38.464 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:22:45.636 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:38.471 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:22:45.650 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:38.472 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:22:45.645 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:38.512 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:22:45.684 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:43.308 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:22:50.487 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:43.311 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:22:50.490 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:43.327 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:22:50.506 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:43.361 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:22:50.533 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:43.362 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:22:50.534 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:43.378 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:22:50.550 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:43.389 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:22:50.568 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:43.394 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:22:50.573 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:43.418 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:22:50.597 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:47.981 INFO:tasks.workunit.client.0.smithi117.stdout: ap list: Running main() from gmock_main.cc 2024-05-15T13:04:47.981 INFO:tasks.workunit.client.0.smithi117.stdout: list: [==========] Running 3 tests from 1 test suite. 2024-05-15T13:04:47.981 INFO:tasks.workunit.client.0.smithi117.stdout: list: [----------] Global test environment set-up. 2024-05-15T13:04:47.981 INFO:tasks.workunit.client.0.smithi117.stdout: list: [----------] 3 tests from NeoradosList 2024-05-15T13:04:47.981 INFO:tasks.workunit.client.0.smithi117.stdout: list: [ RUN ] NeoradosList.ListObjects 2024-05-15T13:04:47.981 INFO:tasks.workunit.client.0.smithi117.stdout: list: [ OK ] NeoradosList.ListObjects (1167787 ms) 2024-05-15T13:04:47.982 INFO:tasks.workunit.client.0.smithi117.stdout: list: [ RUN ] NeoradosList.ListObjectsNS 2024-05-15T13:04:47.982 INFO:tasks.workunit.client.0.smithi117.stdout: list: [ OK ] NeoradosList.ListObjectsNS (6447 ms) 2024-05-15T13:04:47.982 INFO:tasks.workunit.client.0.smithi117.stdout: list: [ RUN ] NeoradosList.ListObjectsMany 2024-05-15T13:04:47.982 INFO:tasks.workunit.client.0.smithi117.stdout: list: [ OK ] NeoradosList.ListObjectsMany (27722 ms) 2024-05-15T13:04:47.982 INFO:tasks.workunit.client.0.smithi117.stdout: list: [----------] 3 tests from NeoradosList (1201956 ms total) 2024-05-15T13:04:47.982 INFO:tasks.workunit.client.0.smithi117.stdout: list: 2024-05-15T13:04:47.982 INFO:tasks.workunit.client.0.smithi117.stdout: list: [----------] Global test environment tear-down 2024-05-15T13:04:47.982 INFO:tasks.workunit.client.0.smithi117.stdout: list: [==========] 3 tests from 1 test suite ran. (1201957 ms total) 2024-05-15T13:04:47.982 INFO:tasks.workunit.client.0.smithi117.stdout: list: [ PASSED ] 3 tests. 2024-05-15T13:04:58.387 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:23:05.566 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:58.390 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:23:05.570 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:58.427 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:23:05.606 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:58.441 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:23:05.613 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:58.445 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:23:05.624 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:58.449 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:23:05.621 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:58.450 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:23:05.629 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:58.479 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:23:05.659 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:58.486 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:23:05.658 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:13.306 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:23:20.485 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:13.311 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:23:20.491 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:13.361 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:23:20.533 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:13.364 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:23:20.536 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:13.369 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:23:20.548 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:13.378 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:23:20.557 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:13.385 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:23:20.564 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:13.404 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:23:20.576 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:13.420 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:23:20.599 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:18.316 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:23:25.495 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:18.318 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:23:25.497 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:18.333 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:23:25.513 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:18.364 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:23:25.537 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:18.367 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:23:25.546 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:18.368 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:23:25.540 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:18.369 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:23:25.548 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:18.385 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:23:25.564 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:18.385 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:23:25.558 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:28.471 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:23:35.650 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:28.532 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:23:35.711 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:28.589 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:23:35.768 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:28.594 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:23:35.766 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:28.599 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:23:35.771 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:28.607 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:23:35.787 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:28.609 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:23:35.789 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:28.632 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:23:35.804 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:28.654 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:23:35.833 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:29.453 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:23:36.625 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:29.454 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:23:36.626 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:29.465 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:23:36.637 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:29.501 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:23:36.680 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:29.502 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:23:36.682 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:29.504 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:23:36.683 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:29.505 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:23:36.684 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:29.513 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:23:36.693 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:29.515 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:23:36.694 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:38.615 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:23:45.794 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:38.617 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:23:45.797 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:38.643 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:23:45.822 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:38.674 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:23:45.846 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:38.676 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:23:45.848 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:38.676 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:23:45.855 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:38.681 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:23:45.860 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:38.701 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:23:45.873 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:38.706 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:23:45.885 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:43.516 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:23:50.695 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:43.518 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:23:50.697 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:43.534 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:23:50.713 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:43.566 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:23:50.738 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:43.567 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:23:50.739 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:43.568 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:23:50.746 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:43.570 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:23:50.749 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:43.581 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:23:50.754 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:43.593 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:23:50.772 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:58.682 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:24:05.861 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:58.684 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:24:05.864 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:58.735 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:24:05.914 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:58.746 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:24:05.919 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:58.749 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:24:05.928 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:58.751 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:24:05.923 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:58.754 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:24:05.933 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:58.786 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:24:05.958 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:58.790 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:24:05.969 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:13.597 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:24:20.776 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:13.652 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:24:20.824 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:13.654 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:24:20.826 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:13.662 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:24:20.841 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:13.689 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:24:20.861 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:13.703 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:24:20.882 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:13.717 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:24:20.896 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:13.723 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:24:20.903 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:13.756 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:24:20.935 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:18.691 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:24:25.870 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:18.693 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:24:25.872 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:18.715 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:24:25.894 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:18.740 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:24:25.912 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:18.743 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:24:25.922 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:18.743 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:24:25.915 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:18.748 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:24:25.927 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:18.761 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:24:25.933 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:18.765 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:24:25.944 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:28.968 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:24:36.140 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:28.968 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:24:36.141 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:28.992 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:24:36.171 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:28.995 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:24:36.167 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:29.001 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:24:36.180 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:29.048 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:24:36.227 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:29.074 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:24:36.253 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:29.084 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:24:36.263 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:29.125 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:24:36.305 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:38.986 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:24:46.165 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:38.988 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:24:46.168 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:39.013 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:24:46.193 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:39.040 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:24:46.213 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:39.045 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:24:46.217 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:39.048 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:24:46.227 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:39.054 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:24:46.233 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:39.070 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:24:46.242 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:39.077 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:24:46.256 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:40.959 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:24:48.138 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:40.971 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:24:48.150 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:40.972 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:24:48.144 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:40.972 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:24:48.152 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:41.012 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:24:48.191 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:41.020 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:24:48.192 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:41.036 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:24:48.215 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:41.039 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:24:48.212 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:41.046 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:24:48.225 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:43.807 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:24:50.986 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:43.856 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:24:51.035 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:43.856 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:24:51.028 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:43.859 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:24:51.031 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:43.866 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:24:51.045 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:43.868 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:24:51.040 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:43.906 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:24:51.085 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:43.907 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:24:51.086 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:43.916 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:24:51.095 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:58.900 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:25:06.079 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:58.901 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:25:06.080 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:58.930 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:25:06.110 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:58.947 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:25:06.119 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:58.948 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:25:06.120 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:58.952 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:25:06.131 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:58.953 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:25:06.133 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:58.976 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:25:06.149 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:58.997 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:25:06.176 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:04.005 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:25:11.184 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:04.008 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:25:11.187 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:04.035 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:25:11.214 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:04.066 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:25:11.245 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:04.070 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:25:11.249 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:04.075 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:25:11.247 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:04.076 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:25:11.249 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:04.087 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:25:11.266 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:04.091 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:25:11.264 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:13.951 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:25:21.130 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:13.954 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:25:21.133 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:13.978 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:25:21.157 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:14.004 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:25:21.176 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:14.014 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:25:21.193 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:14.015 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:25:21.194 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:14.024 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:25:21.197 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:14.032 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:25:21.212 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:14.041 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:25:21.214 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:24.189 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:25:31.368 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:24.197 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:25:31.376 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:24.239 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:25:31.418 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:24.269 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:25:31.441 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:24.270 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:25:31.449 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:24.275 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:25:31.454 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:24.276 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:25:31.448 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:24.310 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:25:31.483 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:24.311 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:25:31.491 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:29.193 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:25:36.372 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:29.197 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:25:36.376 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:29.213 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:25:36.385 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:29.216 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:25:36.389 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:29.221 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:25:36.400 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:29.233 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:25:36.405 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:29.258 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:25:36.437 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:29.266 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:25:36.445 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:29.291 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:25:36.470 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:40.565 INFO:tasks.workunit.client.0.smithi117.stdout:i_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (10055 ms) 2024-05-15T13:07:40.565 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2024-05-15T13:07:40.565 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (13918 ms) 2024-05-15T13:07:40.565 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2024-05-15T13:07:40.565 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (9121 ms) 2024-05-15T13:07:40.565 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2024-05-15T13:07:40.565 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (16936 ms) 2024-05-15T13:07:40.565 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2024-05-15T13:07:40.565 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (10977 ms) 2024-05-15T13:07:40.565 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2024-05-15T13:07:40.565 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (10995 ms) 2024-05-15T13:07:40.566 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2024-05-15T13:07:40.566 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: hmm, no HitSet yet 2024-05-15T13:07:40.566 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: ok, hit_set contains 221:602f83fe:::foo:head 2024-05-15T13:07:40.566 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (10068 ms) 2024-05-15T13:07:40.566 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2024-05-15T13:07:40.566 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: got ls 1715778357,0 2024-05-15T13:07:40.566 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: first is 1715778357 2024-05-15T13:07:40.566 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: got ls 1715778357,0 2024-05-15T13:07:40.566 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: got ls 1715778357,0 2024-05-15T13:07:40.566 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: got ls 1715778357,1715778359,1715778360,0 2024-05-15T13:07:40.566 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: got ls 1715778357,1715778359,1715778360,0 2024-05-15T13:07:40.566 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: got ls 1715778357,1715778359,1715778360,0 2024-05-15T13:07:40.566 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: got ls 1715778357,1715778359,1715778360,1715778362,1715778363,0 2024-05-15T13:07:40.566 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: got ls 1715778357,1715778359,1715778360,1715778362,1715778363,0 2024-05-15T13:07:40.567 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: got ls 1715778357,1715778359,1715778360,1715778362,1715778363,0 2024-05-15T13:07:40.567 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: got ls 1715778357,1715778359,1715778360,1715778362,1715778363,1715778366,1715778367,0 2024-05-15T13:07:40.567 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: got ls 1715778357,1715778359,1715778360,1715778362,1715778363,1715778366,1715778367,0 2024-05-15T13:07:40.567 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: got ls 1715778357,1715778359,1715778360,1715778362,1715778363,1715778366,1715778367,0 2024-05-15T13:07:40.567 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: got ls 1715778360,1715778362,1715778363,1715778366,1715778367,1715778369,1715778370,0 2024-05-15T13:07:40.567 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: first now 1715778360, trimmed 2024-05-15T13:07:40.567 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (22226 ms) 2024-05-15T13:07:40.567 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2024-05-15T13:07:40.567 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: foo0 2024-05-15T13:07:40.567 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-05-15T13:07:40.567 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (17881 ms) 2024-05-15T13:07:40.567 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2024-05-15T13:07:40.567 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (22301 ms) 2024-05-15T13:07:40.568 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2024-05-15T13:07:40.568 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (24619 ms) 2024-05-15T13:07:40.568 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2024-05-15T13:07:40.568 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (7055 ms) 2024-05-15T13:07:40.568 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2024-05-15T13:07:40.568 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (5623 ms) 2024-05-15T13:07:40.568 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2024-05-15T13:07:40.568 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (4210 ms) 2024-05-15T13:07:40.568 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2024-05-15T13:07:40.568 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (4676 ms) 2024-05-15T13:07:40.568 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (284352 ms total) 2024-05-15T13:07:40.568 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: 2024-05-15T13:07:40.568 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [----------] Global test environment tear-down 2024-05-15T13:07:40.568 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (1374830 ms total) 2024-05-15T13:07:40.569 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ PASSED ] 77 tests. 2024-05-15T13:07:40.571 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:07:40.571 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=39899 2024-05-15T13:07:40.571 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 39899 2024-05-15T13:07:40.571 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:07:40.571 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40057 2024-05-15T13:07:40.571 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 40057 2024-05-15T13:07:40.571 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:07:40.571 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40591 2024-05-15T13:07:40.571 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 40591 2024-05-15T13:07:40.571 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:07:40.571 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40361 2024-05-15T13:07:40.571 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 40361 2024-05-15T13:07:40.571 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:07:40.572 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40165 2024-05-15T13:07:40.572 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 40165 2024-05-15T13:07:40.572 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:07:40.572 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=39979 2024-05-15T13:07:40.572 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 39979 2024-05-15T13:07:40.572 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:07:40.572 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40287 2024-05-15T13:07:40.572 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 40287 2024-05-15T13:07:40.572 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:07:40.572 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40766 2024-05-15T13:07:40.572 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 40766 2024-05-15T13:07:40.572 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:07:40.572 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40793 2024-05-15T13:07:40.572 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 40793 2024-05-15T13:07:44.104 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:25:51.283 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:44.105 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:25:51.284 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:44.135 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:25:51.314 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:44.153 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:25:51.332 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:44.153 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:25:51.325 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:44.154 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:25:51.327 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:44.155 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:25:51.334 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:44.185 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:25:51.364 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:44.197 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:25:51.370 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:44.887 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:25:52.059 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:44.888 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:25:52.060 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:44.896 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:25:52.068 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:44.904 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:25:52.083 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:44.905 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:25:52.085 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:44.913 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:25:52.093 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:44.931 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:25:52.110 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:44.932 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:25:52.111 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:44.940 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:25:52.119 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:59.098 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:26:06.277 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:59.101 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:26:06.280 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:59.121 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:26:06.301 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:59.150 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:26:06.322 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:59.151 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:26:06.323 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:59.154 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:26:06.334 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:59.155 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:26:06.335 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:59.170 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:26:06.342 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:59.173 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:26:06.353 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:04.119 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:26:11.298 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:04.122 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:26:11.302 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:04.132 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:26:11.312 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:04.167 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:26:11.339 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:04.168 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:26:11.341 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:04.169 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:26:11.349 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:04.172 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:26:11.352 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:04.178 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:26:11.350 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:04.182 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:26:11.361 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:09.137 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:26:16.316 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:09.143 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:26:16.322 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:09.154 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:26:16.334 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:09.188 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:26:16.360 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:09.191 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:26:16.363 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:09.195 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:26:16.375 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:09.197 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:26:16.376 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:09.202 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:26:16.374 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:09.209 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:26:16.388 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:19.191 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:26:26.370 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:19.195 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:26:26.374 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:19.212 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:26:26.392 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:19.242 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:26:26.415 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:19.244 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:26:26.417 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:19.248 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:26:26.428 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:19.251 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:26:26.431 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:19.259 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:26:26.432 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:19.268 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:26:26.447 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:19.852 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:26:27.025 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:19.853 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:26:27.026 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:19.865 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:26:27.037 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:19.874 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:26:27.053 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:19.875 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:26:27.054 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:19.883 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:26:27.062 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:19.916 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:26:27.096 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:19.918 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:26:27.097 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:19.929 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:26:27.108 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:24.199 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:26:31.378 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:24.204 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:26:31.384 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:24.214 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:26:31.393 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:24.253 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:26:31.432 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:24.254 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:26:31.426 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:24.255 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:26:31.434 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:24.257 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:26:31.430 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:24.265 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:26:31.444 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:24.267 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:26:31.439 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:39.231 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:26:46.410 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:39.236 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:26:46.415 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:39.259 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:26:46.438 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:39.286 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:26:46.458 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:39.287 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:26:46.460 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:39.294 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:26:46.474 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:39.297 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:26:46.476 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:39.303 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:26:46.476 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:39.313 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:26:46.493 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:44.273 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:26:51.452 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:44.277 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:26:51.457 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:44.292 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:26:51.471 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:44.332 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:26:51.511 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:44.333 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:26:51.506 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:44.333 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:26:51.513 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:44.335 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:26:51.507 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:44.343 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:26:51.523 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:44.345 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:26:51.518 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:59.297 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:27:06.476 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:59.298 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:27:06.477 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:59.313 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:27:06.493 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:59.345 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:27:06.518 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:59.346 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:27:06.525 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:59.347 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:27:06.519 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:59.350 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:27:06.530 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:59.365 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:27:06.537 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:59.367 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:27:06.547 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:04.304 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:27:11.483 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:04.307 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:27:11.487 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:04.317 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:27:11.497 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:04.357 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:27:11.529 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:04.358 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:27:11.537 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:04.358 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:27:11.530 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:04.359 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:27:11.539 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:04.368 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:27:11.541 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:04.373 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:27:11.552 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:19.336 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:27:26.515 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:19.340 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:27:26.519 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:19.340 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:27:26.512 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:19.343 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:27:26.516 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:19.357 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:27:26.537 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:19.359 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:27:26.532 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:19.389 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:27:26.569 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:19.391 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:27:26.571 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:19.415 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:27:26.594 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:24.356 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:27:31.535 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:24.357 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:27:31.537 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:24.368 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:27:31.547 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:24.405 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:27:31.577 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:24.407 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:27:31.579 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:24.407 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:27:31.587 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:24.410 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:27:31.590 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:24.416 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:27:31.589 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:24.420 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:27:31.600 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:39.387 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:27:46.566 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:39.389 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:27:46.568 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:39.404 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:27:46.584 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:39.441 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:27:46.620 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:39.442 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:27:46.621 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:39.442 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:27:46.615 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:39.444 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:27:46.616 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:39.460 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:27:46.640 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:39.462 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:27:46.635 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:40.635 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:27:47.814 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:40.644 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:27:47.824 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:40.646 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:27:47.818 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:40.682 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:27:47.862 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:40.692 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:27:47.864 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:40.692 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:27:47.871 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:40.698 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:27:47.878 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:40.707 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:27:47.879 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:40.708 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:27:47.888 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:44.419 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:27:51.598 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:44.424 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:27:51.604 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:44.443 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:27:51.623 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:44.477 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:27:51.650 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:44.482 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:27:51.654 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:44.488 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:27:51.668 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:44.491 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:27:51.671 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:44.495 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:27:51.667 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:44.503 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:27:51.682 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:44.998 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:27:52.170 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:44.998 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:27:52.171 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:45.006 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:27:52.179 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:45.016 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:27:52.195 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:45.017 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:27:52.197 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:45.029 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:27:52.209 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:45.032 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:27:52.212 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:45.034 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:27:52.213 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:45.042 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:27:52.222 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:59.445 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:28:06.624 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:59.449 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:28:06.628 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:59.472 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:28:06.651 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:59.494 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:28:06.667 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:59.501 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:28:06.673 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:59.508 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:28:06.688 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:59.509 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:28:06.689 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:59.518 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:28:06.690 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:59.526 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:28:06.705 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:04.459 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:28:11.638 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:04.464 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:28:11.643 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:04.473 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:28:11.653 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:04.512 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:28:11.691 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:04.514 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:28:11.693 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:04.517 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:28:11.689 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:04.520 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:28:11.692 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:04.524 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:28:11.703 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:04.531 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:28:11.703 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:19.508 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:28:26.686 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:19.512 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:28:26.691 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:19.529 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:28:26.707 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:19.558 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:28:26.730 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:19.559 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:28:26.731 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:19.565 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:28:26.743 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:19.566 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:28:26.745 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:19.575 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:28:26.747 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:19.583 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:28:26.762 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:24.535 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:28:31.713 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:24.586 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:28:31.764 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:24.603 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:28:31.782 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:24.638 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:28:31.809 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:24.640 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:28:31.812 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:24.653 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:28:31.825 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:24.656 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:28:31.835 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:24.659 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:28:31.838 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:24.672 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:28:31.851 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:25.219 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:28:32.391 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:25.220 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:28:32.392 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:25.229 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:28:32.401 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:25.246 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:28:32.425 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:25.247 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:28:32.426 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:25.257 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:28:32.436 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:25.278 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:28:32.457 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:25.279 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:28:32.458 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:25.288 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:28:32.467 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:39.573 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:28:46.751 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:39.577 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:28:46.755 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:39.592 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:28:46.770 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:39.626 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:28:46.797 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:39.627 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:28:46.799 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:39.639 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:28:46.818 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:39.641 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:28:46.819 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:39.643 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:28:46.814 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:39.656 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:28:46.834 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:44.543 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:28:51.720 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:44.592 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:28:51.770 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:44.593 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:28:51.763 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:44.596 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:28:51.767 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:44.609 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:28:51.787 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:44.612 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:28:51.784 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:44.644 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:28:51.822 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:44.645 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:28:51.824 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:44.656 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:28:51.834 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:59.627 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:29:06.805 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:59.630 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:29:06.808 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:59.645 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:29:06.823 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:59.681 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:29:06.859 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:59.682 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:29:06.860 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:59.683 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:29:06.854 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:59.684 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:29:06.855 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:59.699 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:29:06.877 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:59.700 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:29:06.871 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:04.785 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:29:11.962 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:04.786 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:29:11.964 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:04.797 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:29:11.975 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:04.839 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:29:12.009 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:04.839 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:29:12.017 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:04.840 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:29:12.011 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:04.840 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:29:12.018 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:04.850 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:29:12.021 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:04.851 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:29:12.029 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:19.667 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:29:26.844 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:19.715 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:29:26.893 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:19.731 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:29:26.908 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:19.765 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:29:26.943 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:19.767 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:29:26.944 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:19.768 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:29:26.939 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:19.770 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:29:26.940 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:19.784 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:29:26.961 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:19.788 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:29:26.958 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:24.792 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:29:31.970 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:24.795 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:29:31.973 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:24.807 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:29:31.984 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:24.844 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:29:32.015 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:24.845 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:29:32.022 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:24.846 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:29:32.016 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:24.846 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:29:32.023 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:24.856 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:29:32.027 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:24.858 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:29:32.035 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:39.709 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:29:46.886 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:39.759 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:29:46.936 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:39.764 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:29:46.934 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:39.766 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:29:46.936 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:39.776 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:29:46.953 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:39.784 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:29:46.954 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:39.809 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:29:46.986 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:39.812 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:29:46.989 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:39.828 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:29:47.005 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:44.804 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:29:51.981 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:44.807 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:29:51.984 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:44.818 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:29:51.995 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:44.853 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:29:52.022 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:44.854 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:29:52.024 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:44.857 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:29:52.034 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:44.858 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:29:52.035 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:44.864 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:29:52.034 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:44.869 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:29:52.046 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:59.832 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:30:07.008 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:59.836 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:30:07.013 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:59.860 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:30:07.037 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:59.887 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:30:07.064 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:59.889 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:30:07.066 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:59.906 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:30:07.083 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:59.914 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:30:07.084 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:59.915 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:30:07.085 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:59.931 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:30:07.101 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:04.841 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:30:12.018 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:04.845 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:30:12.022 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:04.857 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:30:12.033 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:04.893 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:30:12.063 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:04.893 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:30:12.070 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:04.894 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:30:12.064 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:04.896 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:30:12.073 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:04.904 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:30:12.074 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:04.907 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:30:12.084 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:19.877 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:30:27.053 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:19.881 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:30:27.057 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:19.896 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:30:27.072 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:19.928 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:30:27.097 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:19.929 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:30:27.099 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:19.932 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:30:27.108 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:19.934 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:30:27.110 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:19.947 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:30:27.116 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:19.952 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:30:27.128 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:24.888 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:30:32.065 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:24.891 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:30:32.067 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:24.901 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:30:32.078 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:24.937 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:30:32.106 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:24.938 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:30:32.114 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:24.938 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:30:32.107 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:24.941 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:30:32.117 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:24.948 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:30:32.118 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:24.952 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:30:32.128 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:39.874 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:30:47.050 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:39.925 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:30:47.101 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:39.941 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:30:47.117 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:39.971 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:30:47.140 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:39.973 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:30:47.142 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:39.977 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:30:47.153 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:39.979 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:30:47.155 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:39.988 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:30:47.157 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:40.001 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:30:47.177 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:49.943 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:30:57.119 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:49.945 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:30:57.121 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:49.957 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:30:57.133 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:49.992 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:30:57.160 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:49.993 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:30:57.162 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:49.993 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:30:57.169 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:49.996 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:30:57.172 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:50.008 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:30:57.177 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:50.011 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:30:57.187 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:59.970 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:31:07.145 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:59.972 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:31:07.148 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:59.997 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:31:07.173 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:00.018 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:31:07.187 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:00.019 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:31:07.188 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:00.030 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:31:07.205 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:00.031 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:31:07.207 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:00.035 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:31:07.204 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:00.046 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:31:07.222 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:09.990 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:31:17.165 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:09.991 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:31:17.167 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:10.007 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:31:17.182 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:10.038 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:31:17.206 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:10.039 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:31:17.207 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:10.041 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:31:17.217 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:10.042 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:31:17.218 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:10.052 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:31:17.220 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:10.055 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:31:17.231 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:20.011 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:31:27.186 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:20.013 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:31:27.189 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:20.025 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:31:27.201 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:20.059 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:31:27.227 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:20.060 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:31:27.228 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:20.072 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:31:27.240 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:20.084 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:31:27.260 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:20.086 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:31:27.261 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:20.100 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:31:27.275 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:25.163 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:31:32.338 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:25.168 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:31:32.343 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:25.172 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:31:32.341 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:25.175 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:31:32.344 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:25.178 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:31:32.354 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:25.182 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:31:32.357 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:25.191 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:31:32.366 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:25.195 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:31:32.363 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:25.201 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:31:32.377 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:30.029 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:31:37.204 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:30.033 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:31:37.208 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:30.043 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:31:37.219 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:30.083 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:31:37.259 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:30.085 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:31:37.253 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:30.085 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:31:37.260 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:30.086 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:31:37.255 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:30.094 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:31:37.270 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:30.096 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:31:37.265 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:45.067 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:31:52.242 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:45.070 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:31:52.245 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:45.086 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:31:52.261 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:45.117 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:31:52.285 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:45.118 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:31:52.286 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:45.121 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:31:52.296 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:45.123 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:31:52.298 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:45.134 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:31:52.302 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:45.139 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:31:52.315 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:50.074 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:31:57.249 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:50.081 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:31:57.256 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:50.092 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:31:57.267 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:50.128 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:31:57.303 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:50.129 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:31:57.297 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:50.132 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:31:57.300 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:50.143 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:31:57.311 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:50.146 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:31:57.321 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:50.162 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:31:57.337 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:14:05.110 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:32:12.284 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:14:05.113 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:32:12.288 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:14:05.129 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:32:12.303 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:14:05.162 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:32:12.329 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:14:05.163 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:32:12.331 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:14:05.165 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:32:12.339 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:14:05.166 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:32:12.341 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:14:05.179 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:32:12.347 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:14:05.182 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:32:12.357 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:14:10.125 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:32:17.299 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:14:10.129 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:32:17.304 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:14:10.140 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:32:17.315 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:14:10.179 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:32:17.346 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:14:10.179 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:32:17.347 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:14:10.190 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:32:17.358 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:14:10.193 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:32:17.368 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:14:10.199 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:32:17.374 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:14:10.210 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:32:17.384 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:14:25.173 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:32:32.347 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:14:25.176 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:32:32.350 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:14:25.198 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:32:32.373 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:14:25.225 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:32:32.392 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:14:25.226 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:32:32.393 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:14:25.237 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:32:32.411 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:14:25.242 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:32:32.416 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:14:25.245 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:32:32.413 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:14:25.260 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:32:32.434 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:14:30.140 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:32:37.314 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:14:30.188 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:32:37.362 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:14:30.190 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:32:37.357 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:14:30.193 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:32:37.361 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:14:30.201 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:32:37.375 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:14:30.205 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:32:37.372 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:14:30.265 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:32:37.440 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:14:30.269 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:32:37.444 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:14:30.282 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:32:37.456 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:14:45.173 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:32:52.346 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:14:45.222 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:32:52.389 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:14:45.223 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:32:52.398 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:14:45.224 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:32:52.391 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:14:45.224 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:32:52.398 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:14:45.228 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:32:52.402 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:14:45.242 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:32:52.409 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:14:45.248 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:32:52.422 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:14:45.249 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:32:52.424 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:14:50.229 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:32:57.403 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:14:50.234 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:32:57.408 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:14:50.245 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:32:57.419 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:14:50.280 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:32:57.447 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:14:50.281 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:32:57.455 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:14:50.283 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:32:57.450 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:14:50.285 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:32:57.460 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:14:50.294 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:32:57.461 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:14:50.296 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:32:57.470 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:05.360 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:33:12.534 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:05.366 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:33:12.540 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:05.406 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:33:12.579 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:05.431 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:33:12.598 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:05.433 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:33:12.607 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:05.435 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:33:12.602 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:05.437 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:33:12.610 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:05.459 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:33:12.626 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:05.467 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:33:12.640 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:08.385 INFO:tasks.workunit.client.0.smithi117.stdout: ec_io: Running main() from gmock_main.cc 2024-05-15T13:15:08.386 INFO:tasks.workunit.client.0.smithi117.stdout: ec_io: [==========] Running 2 tests from 1 test suite. 2024-05-15T13:15:08.386 INFO:tasks.workunit.client.0.smithi117.stdout: ec_io: [----------] Global test environment set-up. 2024-05-15T13:15:08.386 INFO:tasks.workunit.client.0.smithi117.stdout: ec_io: [----------] 2 tests from NeoRadosECIo 2024-05-15T13:15:08.386 INFO:tasks.workunit.client.0.smithi117.stdout: ec_io: [ RUN ] NeoRadosECIo.SimpleWrite 2024-05-15T13:15:08.386 INFO:tasks.workunit.client.0.smithi117.stdout: ec_io: [ OK ] NeoRadosECIo.SimpleWrite (11571 ms) 2024-05-15T13:15:08.386 INFO:tasks.workunit.client.0.smithi117.stdout: ec_io: [ RUN ] NeoRadosECIo.ReadOp 2024-05-15T13:15:08.386 INFO:tasks.workunit.client.0.smithi117.stdout: ec_io: [ OK ] NeoRadosECIo.ReadOp (1810769 ms) 2024-05-15T13:15:08.386 INFO:tasks.workunit.client.0.smithi117.stdout: ec_io: [----------] 2 tests from NeoRadosECIo (1822340 ms total) 2024-05-15T13:15:08.386 INFO:tasks.workunit.client.0.smithi117.stdout: ec_io: 2024-05-15T13:15:08.386 INFO:tasks.workunit.client.0.smithi117.stdout: ec_io: [----------] Global test environment tear-down 2024-05-15T13:15:08.386 INFO:tasks.workunit.client.0.smithi117.stdout: ec_io: [==========] 2 tests from 1 test suite ran. (1822340 ms total) 2024-05-15T13:15:08.386 INFO:tasks.workunit.client.0.smithi117.stdout: ec_io: [ PASSED ] 2 tests. 2024-05-15T13:15:08.387 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:15:08.387 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=39999 2024-05-15T13:15:08.387 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 39999 2024-05-15T13:15:08.387 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:15:08.387 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40624 2024-05-15T13:15:08.387 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 40624 2024-05-15T13:15:08.387 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:15:08.388 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40517 2024-05-15T13:15:08.388 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 40517 2024-05-15T13:15:08.388 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:15:08.388 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=39874 2024-05-15T13:15:08.388 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 39874 2024-05-15T13:15:08.388 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:15:08.388 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=39937 2024-05-15T13:15:08.388 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 39937 2024-05-15T13:15:08.388 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:15:08.388 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40545 2024-05-15T13:15:08.388 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 40545 2024-05-15T13:15:08.388 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:15:08.388 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=39865 2024-05-15T13:15:08.388 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 39865 2024-05-15T13:15:10.263 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:33:17.437 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:10.312 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:33:17.485 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:10.316 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:33:17.490 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:10.328 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:33:17.502 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:10.332 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:33:17.505 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:10.350 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:33:17.524 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:10.363 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:33:17.530 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:10.364 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:33:17.531 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:10.378 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:33:17.545 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:20.614 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:33:27.788 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:20.647 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:33:27.820 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:20.679 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:33:27.852 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:20.692 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:33:27.859 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:20.699 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:33:27.866 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:20.714 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:33:27.887 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:20.717 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:33:27.891 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:20.729 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:33:27.896 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:20.752 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:33:27.926 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:30.385 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:33:37.557 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:30.452 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:33:37.625 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:30.488 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:33:37.662 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:30.512 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:33:37.678 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:30.514 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:33:37.688 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:30.517 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:33:37.684 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:30.522 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:33:37.695 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:30.546 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:33:37.712 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:30.559 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:33:37.733 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:35.623 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:33:42.789 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:35.626 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:33:42.799 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:35.627 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:33:42.793 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:35.678 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:33:42.851 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:35.703 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:33:42.876 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:35.704 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:33:42.876 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:35.741 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:33:42.914 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:35.755 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:33:42.928 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:35.762 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:33:42.929 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:37.914 INFO:tasks.workunit.client.0.smithi117.stdout: misc: Running main() from gmock_main.cc 2024-05-15T13:15:37.915 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [==========] Running 12 tests from 1 test suite. 2024-05-15T13:15:37.915 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [----------] Global test environment set-up. 2024-05-15T13:15:37.915 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [----------] 12 tests from NeoRadosMisc 2024-05-15T13:15:37.915 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [ RUN ] NeoRadosMisc.Version 2024-05-15T13:15:37.915 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [ OK ] NeoRadosMisc.Version (4388 ms) 2024-05-15T13:15:37.915 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [ RUN ] NeoRadosMisc.WaitOSDMap 2024-05-15T13:15:37.916 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [ OK ] NeoRadosMisc.WaitOSDMap (5885 ms) 2024-05-15T13:15:37.916 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [ RUN ] NeoRadosMisc.LongName 2024-05-15T13:15:37.916 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [ OK ] NeoRadosMisc.LongName (1805471 ms) 2024-05-15T13:15:37.916 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [ RUN ] NeoRadosMisc.LongLocator 2024-05-15T13:15:37.916 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [ OK ] NeoRadosMisc.LongLocator (4463 ms) 2024-05-15T13:15:37.916 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [ RUN ] NeoRadosMisc.LongNamespace 2024-05-15T13:15:37.916 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [ OK ] NeoRadosMisc.LongNamespace (4797 ms) 2024-05-15T13:15:37.916 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [ RUN ] NeoRadosMisc.LongAttrName 2024-05-15T13:15:37.917 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [ OK ] NeoRadosMisc.LongAttrName (3331 ms) 2024-05-15T13:15:37.917 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [ RUN ] NeoRadosMisc.Exec 2024-05-15T13:15:37.917 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [ OK ] NeoRadosMisc.Exec (3396 ms) 2024-05-15T13:15:37.917 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [ RUN ] NeoRadosMisc.Operate1 2024-05-15T13:15:37.917 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [ OK ] NeoRadosMisc.Operate1 (4755 ms) 2024-05-15T13:15:37.917 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [ RUN ] NeoRadosMisc.Operate2 2024-05-15T13:15:37.917 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [ OK ] NeoRadosMisc.Operate2 (3948 ms) 2024-05-15T13:15:37.917 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [ RUN ] NeoRadosMisc.BigObject 2024-05-15T13:15:37.918 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [ OK ] NeoRadosMisc.BigObject (6581 ms) 2024-05-15T13:15:37.918 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [ RUN ] NeoRadosMisc.BigAttr 2024-05-15T13:15:37.918 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [ OK ] NeoRadosMisc.BigAttr (2085 ms) 2024-05-15T13:15:37.918 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [ RUN ] NeoRadosMisc.WriteSame 2024-05-15T13:15:37.918 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [ OK ] NeoRadosMisc.WriteSame (2750 ms) 2024-05-15T13:15:37.918 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [----------] 12 tests from NeoRadosMisc (1851851 ms total) 2024-05-15T13:15:37.918 INFO:tasks.workunit.client.0.smithi117.stdout: misc: 2024-05-15T13:15:37.919 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [----------] Global test environment tear-down 2024-05-15T13:15:37.919 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [==========] 12 tests from 1 test suite ran. (1851851 ms total) 2024-05-15T13:15:37.919 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [ PASSED ] 12 tests. 2024-05-15T13:15:50.508 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:33:57.681 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:50.573 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:33:57.745 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:50.639 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:33:57.805 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:50.639 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:33:57.812 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:50.641 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:33:57.807 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:50.643 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:33:57.816 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:50.656 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:33:57.828 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:50.677 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:33:57.850 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:50.682 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:33:57.848 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:51.534 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:33:58.700 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:51.535 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:33:58.702 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:51.547 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:33:58.713 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:51.842 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:33:59.015 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:51.843 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:33:59.016 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:51.858 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:33:59.031 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:52.003 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:33:59.175 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:52.004 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:33:59.177 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:15:52.016 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:33:59.189 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:05.635 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:34:12.808 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:05.637 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:34:12.810 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:05.669 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:34:12.842 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:05.690 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:34:12.856 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:05.695 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:34:12.861 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:05.703 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:34:12.876 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:05.707 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:34:12.880 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:05.725 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:34:12.891 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:05.741 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:34:12.914 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:07.465 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:34:14.638 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:07.478 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:34:14.651 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:07.493 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:34:14.666 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:07.494 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:34:14.660 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:07.498 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:34:14.664 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:07.550 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:34:14.723 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:07.568 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:34:14.741 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:07.578 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:34:14.744 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:07.597 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:34:14.769 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:10.645 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:34:17.818 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:10.647 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:34:17.813 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:10.650 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:34:17.823 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:10.651 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:34:17.817 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:10.663 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:34:17.829 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:10.663 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:34:17.836 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:10.722 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:34:17.894 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:10.727 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:34:17.899 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:10.744 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:34:17.917 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:20.847 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:34:28.019 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:20.919 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:34:28.092 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:20.952 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:34:28.125 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:20.987 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:34:28.159 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:20.992 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:34:28.157 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:21.001 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:34:28.174 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:21.002 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:34:28.168 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:21.041 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:34:28.207 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:21.043 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:34:28.216 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:30.898 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:34:38.071 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:30.964 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:34:38.129 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:30.967 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:34:38.139 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:30.976 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:34:38.142 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:31.010 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:34:38.176 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:31.034 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:34:38.206 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:31.036 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:34:38.209 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:31.039 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:34:38.211 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:31.076 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:34:38.249 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:32.015 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: Running main() from gmock_main.cc 2024-05-15T13:16:32.015 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2024-05-15T13:16:32.016 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [----------] Global test environment set-up. 2024-05-15T13:16:32.016 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [----------] 26 tests from LibRadosAio 2024-05-15T13:16:32.016 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2024-05-15T13:16:32.016 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAio.TooBig (6518 ms) 2024-05-15T13:16:32.016 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2024-05-15T13:16:32.016 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (1807610 ms) 2024-05-15T13:16:32.016 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2024-05-15T13:16:32.016 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (4169 ms) 2024-05-15T13:16:32.016 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2024-05-15T13:16:32.016 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (4544 ms) 2024-05-15T13:16:32.016 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2024-05-15T13:16:32.016 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (4697 ms) 2024-05-15T13:16:32.016 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2024-05-15T13:16:32.016 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (3567 ms) 2024-05-15T13:16:32.016 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2024-05-15T13:16:32.017 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (3249 ms) 2024-05-15T13:16:32.017 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2024-05-15T13:16:32.017 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (2632 ms) 2024-05-15T13:16:32.017 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2024-05-15T13:16:32.017 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (3933 ms) 2024-05-15T13:16:32.017 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2024-05-15T13:16:32.017 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (4548 ms) 2024-05-15T13:16:32.017 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2024-05-15T13:16:32.017 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (4702 ms) 2024-05-15T13:16:32.017 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2024-05-15T13:16:32.017 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (3234 ms) 2024-05-15T13:16:32.017 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2024-05-15T13:16:32.017 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (4113 ms) 2024-05-15T13:16:32.017 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2024-05-15T13:16:32.017 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (3608 ms) 2024-05-15T13:16:32.017 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2024-05-15T13:16:32.018 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAio.Flush (3379 ms) 2024-05-15T13:16:32.018 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2024-05-15T13:16:32.018 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (4694 ms) 2024-05-15T13:16:32.018 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2024-05-15T13:16:32.018 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (3701 ms) 2024-05-15T13:16:32.018 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2024-05-15T13:16:32.018 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (4428 ms) 2024-05-15T13:16:32.018 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2024-05-15T13:16:32.018 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (3543 ms) 2024-05-15T13:16:32.018 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2024-05-15T13:16:32.018 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (3261 ms) 2024-05-15T13:16:32.018 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2024-05-15T13:16:32.018 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (3801 ms) 2024-05-15T13:16:32.018 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2024-05-15T13:16:32.018 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (3409 ms) 2024-05-15T13:16:32.018 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2024-05-15T13:16:32.019 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (3144 ms) 2024-05-15T13:16:32.019 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2024-05-15T13:16:32.019 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (2844 ms) 2024-05-15T13:16:32.019 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2024-05-15T13:16:32.019 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (4502 ms) 2024-05-15T13:16:32.019 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2024-05-15T13:16:32.019 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (4626 ms) 2024-05-15T13:16:32.019 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [----------] 26 tests from LibRadosAio (1906456 ms total) 2024-05-15T13:16:32.019 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: 2024-05-15T13:16:32.019 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2024-05-15T13:16:35.937 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:34:43.109 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:36.001 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:34:43.166 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:36.002 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:34:43.168 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:36.019 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:34:43.191 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:36.021 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:34:43.187 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:36.023 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:34:43.196 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:36.024 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:34:43.196 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:36.051 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:34:43.223 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:36.069 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:34:43.241 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:40.795 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:34:47.967 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:40.845 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:34:48.017 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:40.862 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:34:48.034 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:40.911 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:34:48.083 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:40.912 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:34:48.085 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:40.914 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:34:48.080 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:40.916 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:34:48.082 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:40.934 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:34:48.099 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:40.937 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:34:48.109 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:50.871 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:34:58.043 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:50.930 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:34:58.102 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:50.968 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:34:58.140 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:50.979 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:34:58.144 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:50.983 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:34:58.148 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:50.999 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:34:58.172 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:51.001 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:34:58.173 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:51.008 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:34:58.173 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:16:51.027 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:34:58.199 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:17:03.676 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleWr api_aio_pp: Running main() from gmock_main.cc 2024-05-15T13:17:03.676 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2024-05-15T13:17:03.676 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [----------] Global test environment set-up. 2024-05-15T13:17:03.676 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2024-05-15T13:17:03.676 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2024-05-15T13:17:03.676 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (6475 ms) 2024-05-15T13:17:03.676 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2024-05-15T13:17:03.676 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (1836513 ms) 2024-05-15T13:17:03.676 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2024-05-15T13:17:03.676 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (9353 ms) 2024-05-15T13:17:03.676 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2024-05-15T13:17:03.676 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (3103 ms) 2024-05-15T13:17:03.677 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2024-05-15T13:17:03.677 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (4127 ms) 2024-05-15T13:17:03.677 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2024-05-15T13:17:03.677 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (3834 ms) 2024-05-15T13:17:03.677 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2024-05-15T13:17:03.677 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (3674 ms) 2024-05-15T13:17:03.677 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-05-15T13:17:03.677 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (3445 ms) 2024-05-15T13:17:03.677 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2024-05-15T13:17:03.677 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (4543 ms) 2024-05-15T13:17:03.677 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2024-05-15T13:17:03.677 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (3272 ms) 2024-05-15T13:17:03.677 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2024-05-15T13:17:03.677 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (3710 ms) 2024-05-15T13:17:03.678 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2024-05-15T13:17:03.678 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (3672 ms) 2024-05-15T13:17:03.678 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2024-05-15T13:17:03.678 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (3380 ms) 2024-05-15T13:17:03.678 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-05-15T13:17:03.678 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (4356 ms) 2024-05-15T13:17:03.678 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-05-15T13:17:03.678 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (2895 ms) 2024-05-15T13:17:03.678 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-05-15T13:17:03.678 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (3267 ms) 2024-05-15T13:17:03.678 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-05-15T13:17:03.678 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (4437 ms) 2024-05-15T13:17:03.678 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2024-05-15T13:17:03.678 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (4506 ms) 2024-05-15T13:17:03.679 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2024-05-15T13:17:03.679 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (2913 ms) 2024-05-15T13:17:03.679 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2024-05-15T13:17:03.679 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (3197 ms) 2024-05-15T13:17:03.679 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2024-05-15T13:17:03.679 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (3357 ms) 2024-05-15T13:17:03.679 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2024-05-15T13:17:03.679 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (3538 ms) 2024-05-15T13:17:03.679 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2024-05-15T13:17:03.679 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (3349 ms) 2024-05-15T13:17:03.679 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2024-05-15T13:17:03.679 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (3415 ms) 2024-05-15T13:17:03.679 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2024-05-15T13:17:03.679 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (3194 ms) 2024-05-15T13:17:03.679 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2024-05-15T13:17:03.680 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (3061 ms) 2024-05-15T13:17:03.680 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2024-05-15T13:17:06.198 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:35:13.370 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:17:06.205 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:35:13.377 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:17:06.249 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:35:13.421 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:17:06.266 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:35:13.438 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:17:06.273 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:35:13.438 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:17:06.280 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:35:13.453 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:17:06.292 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:35:13.457 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:17:06.315 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:35:13.487 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:17:06.349 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:35:13.514 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:17:16.208 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:35:23.379 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:17:16.262 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:35:23.426 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:17:16.267 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:35:23.432 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:17:16.273 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:35:23.445 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:17:16.303 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:35:23.475 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:17:16.321 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:35:23.486 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:17:16.328 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:35:23.500 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:17:16.366 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:35:23.538 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:17:16.422 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:35:23.594 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:17:21.071 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:35:28.243 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:17:21.073 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:35:28.245 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:17:21.095 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:35:28.266 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:17:21.121 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:35:28.293 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:17:21.122 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:35:28.287 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:17:21.123 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:35:28.295 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:17:21.123 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:35:28.288 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:17:21.140 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:35:28.312 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:17:21.143 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:35:28.308 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:17:36.192 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:35:43.364 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:17:36.204 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:35:43.376 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:17:36.208 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:35:43.372 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:17:36.224 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:35:43.396 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:17:36.230 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:35:43.402 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:17:36.274 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:35:43.439 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:17:36.322 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:35:43.487 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:17:36.332 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:35:43.504 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:17:36.393 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:35:43.565 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:17:51.311 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:35:58.482 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:17:51.378 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:35:58.549 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:17:51.391 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:35:58.555 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:17:51.398 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:35:58.563 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:17:51.441 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:35:58.613 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:17:51.442 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:35:58.606 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:17:51.448 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:35:58.620 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:17:51.451 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:35:58.623 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:17:51.497 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:35:58.669 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:17:56.193 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:36:03.364 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:17:56.197 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:36:03.368 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:17:56.217 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:36:03.388 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:17:56.242 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:36:03.406 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:17:56.243 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:36:03.408 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:17:56.256 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:36:03.421 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:17:56.265 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:36:03.436 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:17:56.272 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:36:03.443 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:17:56.292 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:36:03.463 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:18:04.578 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:36:11.749 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:18:04.580 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:36:11.751 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:18:04.608 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:36:11.780 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:18:04.630 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:36:11.795 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:18:04.632 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:36:11.796 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:18:04.641 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:36:11.813 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:18:04.645 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:36:11.816 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:18:04.659 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:36:11.824 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:18:04.672 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:36:11.843 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:18:06.300 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:36:13.471 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:18:06.302 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:36:13.473 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:18:06.313 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:36:13.484 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:18:06.346 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:36:13.511 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:18:06.347 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:36:13.512 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:18:06.351 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:36:13.523 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:18:06.353 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:36:13.525 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:18:06.358 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:36:13.523 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:18:06.385 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:36:13.557 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:18:16.451 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:36:23.622 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:18:16.453 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:36:23.624 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:18:16.477 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:36:23.649 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:18:16.502 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:36:23.666 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:18:16.503 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:36:23.667 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:18:16.503 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:36:23.675 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:18:16.510 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:36:23.681 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:18:16.526 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:36:23.691 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:18:16.539 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:36:23.711 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:18:32.812 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAite 2024-05-15T13:18:32.812 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (7109 ms) 2024-05-15T13:18:32.812 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2024-05-15T13:18:32.812 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (7944 ms) 2024-05-15T13:18:32.812 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2024-05-15T13:18:32.812 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (7971 ms) 2024-05-15T13:18:32.812 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2024-05-15T13:18:32.812 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (7565 ms) 2024-05-15T13:18:32.813 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2024-05-15T13:18:32.813 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (7686 ms) 2024-05-15T13:18:32.813 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2024-05-15T13:18:32.813 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (6892 ms) 2024-05-15T13:18:32.813 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2024-05-15T13:18:32.813 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (7287 ms) 2024-05-15T13:18:32.813 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2024-05-15T13:18:32.813 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (7307 ms) 2024-05-15T13:18:32.813 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2024-05-15T13:18:32.813 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (7897 ms) 2024-05-15T13:18:32.813 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2024-05-15T13:18:32.813 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (7971 ms) 2024-05-15T13:18:32.813 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-05-15T13:18:32.813 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (7856 ms) 2024-05-15T13:18:32.813 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2024-05-15T13:18:32.814 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (7922 ms) 2024-05-15T13:18:32.814 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2024-05-15T13:18:32.814 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (7534 ms) 2024-05-15T13:18:32.814 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2024-05-15T13:18:32.814 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (7138 ms) 2024-05-15T13:18:32.814 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2024-05-15T13:18:32.814 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (7632 ms) 2024-05-15T13:18:32.814 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2024-05-15T13:18:32.814 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (7082 ms) 2024-05-15T13:18:32.814 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (120793 ms total) 2024-05-15T13:18:32.814 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: 2024-05-15T13:18:32.814 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [----------] Global test environment tear-down 2024-05-15T13:18:32.814 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (2027249 ms total) 2024-05-15T13:18:32.814 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ PASSED ] 42 tests. 2024-05-15T13:18:36.553 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:36:43.724 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:18:36.558 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:36:43.730 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:18:36.563 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:36:43.727 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:18:36.568 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:36:43.732 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:18:36.621 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:36:43.792 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:18:36.621 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:36:43.785 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:18:36.639 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:36:43.810 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:18:36.646 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:36:43.817 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:18:36.707 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:36:43.878 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:18:41.400 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:36:48.571 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:18:41.401 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:36:48.572 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:18:41.416 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:36:48.587 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:18:41.448 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:36:48.612 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:18:41.449 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:36:48.613 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:18:41.451 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:36:48.622 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:18:41.452 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:36:48.623 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:18:41.463 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:36:48.627 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:18:41.467 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:36:48.638 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:18:56.561 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:37:03.731 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:18:56.566 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:37:03.730 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:18:56.567 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:37:03.738 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:18:56.569 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:37:03.740 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:18:56.609 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:37:03.780 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:18:56.621 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:37:03.785 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:18:56.643 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:37:03.814 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:18:56.648 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:37:03.819 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:18:56.663 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:37:03.827 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:11.542 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:37:18.713 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:11.544 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:37:18.715 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:11.572 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:37:18.743 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:11.596 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:37:18.760 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:11.600 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:37:18.763 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:11.600 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:37:18.771 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:11.602 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:37:18.773 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:11.626 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:37:18.790 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:11.630 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:37:18.801 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:16.705 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:37:23.868 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:16.739 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:37:23.909 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:16.746 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:37:23.917 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:16.747 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:37:23.918 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:16.760 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:37:23.924 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:16.761 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:37:23.932 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:16.786 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:37:23.949 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:16.799 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:37:23.969 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:16.823 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:37:23.994 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:26.631 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:37:33.802 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:26.633 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:37:33.804 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:26.662 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:37:33.832 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:26.682 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:37:33.846 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:26.685 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:37:33.856 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:26.686 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:37:33.850 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:26.691 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:37:33.862 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:26.706 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:37:33.870 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:26.711 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:37:33.882 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:36.676 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:37:43.847 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:36.678 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:37:43.849 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:36.700 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:37:43.871 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:36.724 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:37:43.888 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:36.726 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:37:43.889 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:36.731 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:37:43.901 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:36.733 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:37:43.904 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:36.747 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:37:43.911 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:36.767 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:37:43.937 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:39.574 INFO:tasks.workunit.client.0.smithi117.stdout:io.RoundTripAppendPP (3524 ms) 2024-05-15T13:19:39.574 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2024-05-15T13:19:39.574 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (3562 ms) 2024-05-15T13:19:39.574 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-05-15T13:19:39.574 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (3061 ms) 2024-05-15T13:19:39.574 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-05-15T13:19:39.577 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (3337 ms) 2024-05-15T13:19:39.577 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2024-05-15T13:19:39.577 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: setting pool EIO 2024-05-15T13:19:39.577 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: max_success 5, min_failed 6 2024-05-15T13:19:39.577 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (6668 ms) 2024-05-15T13:19:39.577 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2024-05-15T13:19:39.577 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (4180 ms) 2024-05-15T13:19:39.578 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (1958918 ms total) 2024-05-15T13:19:39.578 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: 2024-05-15T13:19:39.578 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2024-05-15T13:19:39.578 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-05-15T13:19:39.578 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (2667 ms) 2024-05-15T13:19:39.578 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-05-15T13:19:39.578 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (9038 ms) 2024-05-15T13:19:39.578 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2024-05-15T13:19:39.578 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (16105 ms) 2024-05-15T13:19:39.578 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2024-05-15T13:19:39.578 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3196 ms) 2024-05-15T13:19:39.578 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (31006 ms total) 2024-05-15T13:19:39.578 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: 2024-05-15T13:19:39.578 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2024-05-15T13:19:39.579 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-05-15T13:19:39.579 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (3440 ms) 2024-05-15T13:19:39.579 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (3440 ms total) 2024-05-15T13:19:39.579 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: 2024-05-15T13:19:39.579 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2024-05-15T13:19:39.579 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2024-05-15T13:19:39.579 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (13479 ms) 2024-05-15T13:19:39.579 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2024-05-15T13:19:39.579 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (7981 ms) 2024-05-15T13:19:39.579 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2024-05-15T13:19:39.579 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (7022 ms) 2024-05-15T13:19:39.579 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2024-05-15T13:19:39.579 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (7431 ms) 2024-05-15T13:19:39.579 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2024-05-15T13:19:39.579 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (2760 ms) 2024-05-15T13:19:39.580 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-05-15T13:19:39.580 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (7583 ms) 2024-05-15T13:19:39.580 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-05-15T13:19:39.580 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (6909 ms) 2024-05-15T13:19:39.580 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2024-05-15T13:19:39.580 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (7604 ms) 2024-05-15T13:19:39.580 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2024-05-15T13:19:39.580 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (6869 ms) 2024-05-15T13:19:39.580 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2024-05-15T13:19:39.580 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (7709 ms) 2024-05-15T13:19:39.580 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2024-05-15T13:19:39.580 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (6935 ms) 2024-05-15T13:19:39.580 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2024-05-15T13:19:39.580 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (7942 ms) 2024-05-15T13:19:39.580 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-05-15T13:19:39.580 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (6861 ms) 2024-05-15T13:19:39.581 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP2 2024-05-15T13:19:41.803 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:37:48.973 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:41.805 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:37:48.975 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:41.821 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:37:48.990 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:41.852 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:37:49.015 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:41.853 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:37:49.017 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:41.867 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:37:49.030 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:41.870 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:37:49.041 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:41.871 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:37:49.042 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:41.886 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:37:49.057 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:56.825 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:38:03.995 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:56.898 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:38:04.069 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:56.934 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:38:04.104 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:56.957 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:38:04.120 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:56.962 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:38:04.125 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:56.968 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:38:04.139 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:56.977 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:38:04.147 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:56.993 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:38:04.157 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:57.010 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:38:04.181 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:59.163 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:38:06.333 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:59.166 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:38:06.336 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:59.181 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:38:06.352 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:59.227 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:38:06.391 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:59.228 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:38:06.392 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:59.232 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:38:06.402 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:59.235 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:38:06.406 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:59.245 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:38:06.408 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:19:59.251 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:38:06.421 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:20:11.941 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:38:19.111 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:20:11.946 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:38:19.117 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:20:11.979 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:38:19.149 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:20:12.004 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:38:19.174 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:20:12.009 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:38:19.172 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:20:12.010 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:38:19.181 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:20:12.012 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:38:19.175 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:20:12.038 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:38:19.201 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:20:12.039 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:38:19.210 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:20:16.824 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:38:23.994 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:20:16.826 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:38:23.997 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:20:16.840 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:38:24.010 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:20:16.871 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:38:24.034 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:20:16.872 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:38:24.036 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:20:16.881 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:38:24.051 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:20:16.882 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:38:24.052 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:20:16.885 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:38:24.049 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:20:16.895 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:38:24.066 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:20:24.118 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (3559 ms) 2024-05-15T13:20:24.118 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2024-05-15T13:20:24.118 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (7243 ms) 2024-05-15T13:20:24.118 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-05-15T13:20:24.118 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (7602 ms) 2024-05-15T13:20:24.118 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2024-05-15T13:20:24.118 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (6919 ms) 2024-05-15T13:20:24.118 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2024-05-15T13:20:24.118 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (7933 ms) 2024-05-15T13:20:24.119 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2024-05-15T13:20:24.119 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (7014 ms) 2024-05-15T13:20:24.119 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2024-05-15T13:20:24.119 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (7829 ms) 2024-05-15T13:20:24.119 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (145184 ms total) 2024-05-15T13:20:24.119 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: 2024-05-15T13:20:24.119 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [----------] Global test environment tear-down 2024-05-15T13:20:24.119 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (2138548 ms total) 2024-05-15T13:20:24.119 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ PASSED ] 57 tests. 2024-05-15T13:20:24.124 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:20:24.124 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=39886 2024-05-15T13:20:24.124 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 39886 2024-05-15T13:20:24.124 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:20:24.124 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40872 2024-05-15T13:20:24.124 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 40872 2024-05-15T13:20:24.124 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:20:24.125 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40078 2024-05-15T13:20:24.125 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 40078 2024-05-15T13:20:24.125 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:20:24.125 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40952 2024-05-15T13:20:24.125 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 40952 2024-05-15T13:20:24.125 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:20:24.125 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40200 2024-05-15T13:20:24.125 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 40200 2024-05-15T13:20:24.125 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:20:24.125 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40689 2024-05-15T13:20:24.125 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 40689 2024-05-15T13:20:24.125 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:20:24.125 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=41014 2024-05-15T13:20:24.125 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 41014 2024-05-15T13:20:24.125 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:20:24.126 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40096 2024-05-15T13:20:24.126 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 40096 2024-05-15T13:20:24.126 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:20:24.126 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40574 2024-05-15T13:20:24.126 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 40574 2024-05-15T13:20:24.126 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:20:24.126 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40840 2024-05-15T13:20:24.126 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 40840 2024-05-15T13:20:24.126 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:20:24.126 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40671 2024-05-15T13:20:24.126 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 40671 2024-05-15T13:20:24.126 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:20:24.126 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40894 2024-05-15T13:20:24.126 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 40894 2024-05-15T13:20:24.127 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:20:24.127 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40470 2024-05-15T13:20:24.127 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 40470 2024-05-15T13:20:24.127 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:20:24.127 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40039 2024-05-15T13:20:24.127 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 40039 2024-05-15T13:20:24.127 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:20:24.127 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40993 2024-05-15T13:20:24.127 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 40993 2024-05-15T13:20:24.127 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:20:24.127 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40381 2024-05-15T13:20:24.127 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 40381 2024-05-15T13:20:24.127 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:20:24.127 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=39858 2024-05-15T13:20:24.127 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 39858 2024-05-15T13:20:24.128 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:20:24.128 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=39916 2024-05-15T13:20:24.128 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 39916 2024-05-15T13:20:24.128 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:20:24.128 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40312 2024-05-15T13:20:24.128 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 40312 2024-05-15T13:20:24.128 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:20:24.128 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=39954 2024-05-15T13:20:24.128 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 39954 2024-05-15T13:20:24.128 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:20:24.128 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40403 2024-05-15T13:20:24.128 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 40403 2024-05-15T13:20:24.128 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:20:24.128 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40456 2024-05-15T13:20:24.128 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 40456 2024-05-15T13:20:24.128 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:20:24.129 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=40807 2024-05-15T13:20:24.129 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 40807 2024-05-15T13:20:24.129 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:20:24.129 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=41060 2024-05-15T13:20:24.129 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 41060 2024-05-15T13:20:24.129 INFO:tasks.workunit.client.0.smithi117.stderr:+ exit 0 2024-05-15T13:20:24.129 INFO:tasks.workunit.client.0.smithi117.stderr:+ cleanup 2024-05-15T13:20:24.129 INFO:tasks.workunit.client.0.smithi117.stderr:+ pkill -P 39852 2024-05-15T13:20:24.135 INFO:tasks.workunit.client.0.smithi117.stderr:+ true 2024-05-15T13:20:24.136 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-05-15T13:20:24.136 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-05-15T13:20:24.164 INFO:tasks.workunit:Stopping ['rados/test.sh'] on client.0... 2024-05-15T13:20:24.164 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-05-15T13:20:24.490 DEBUG:teuthology.parallel:result is None 2024-05-15T13:20:24.491 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-05-15T13:20:24.516 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-05-15T13:20:24.516 DEBUG:teuthology.orchestra.run.smithi117:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-05-15T13:20:24.577 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-05-15T13:20:24.577 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-05-15T13:20:24.588 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-05-15T13:20:24.588 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-15T13:20:24.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.719+0000 7f0ad6dc6640 10 monclient: get_monmap_and_config 2024-05-15T13:20:24.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.719+0000 7f0ad6dc6640 10 monclient: build_initial_monmap 2024-05-15T13:20:24.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.719+0000 7f0ad6dc6640 10 monclient: monmap: 2024-05-15T13:20:24.718 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T13:20:24.718 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:24.718 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T13:20:24.720284+0000 2024-05-15T13:20:24.718 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T13:20:24.720284+0000 2024-05-15T13:20:24.718 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T13:20:24.718 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:24.719 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T13:20:24.719 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T13:20:24.719 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T13:20:24.719 INFO:teuthology.orchestra.run.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T13:20:24.719 INFO:teuthology.orchestra.run.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T13:20:24.719 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:24.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.720+0000 7f0ad6dc6640 1 Processor -- start 2024-05-15T13:20:24.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.720+0000 7f0ad6dc6640 1 -- start start 2024-05-15T13:20:24.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.720+0000 7f0ad6dc6640 10 monclient: init 2024-05-15T13:20:24.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.720+0000 7f0ad6dc6640 10 monclient: _reopen_session rank -1 2024-05-15T13:20:24.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.720+0000 7f0ad6dc6640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-05-15T13:20:24.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.720+0000 7f0ad6dc6640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0ad006bd70 0x7f0ad0142940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:24.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.720+0000 7f0ad6dc6640 10 monclient(hunting): picked mon.noname-b con 0x7f0ad006bd70 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T13:20:24.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.720+0000 7f0ad6dc6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0ad0070b80 addr v1:172.21.15.117:6789/0 2024-05-15T13:20:24.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.720+0000 7f0ad6dc6640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f0ad00ad820 0x7f0ad00adcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:24.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.720+0000 7f0ad6dc6640 10 monclient(hunting): picked mon.noname-a con 0x7f0ad00ad820 addr v2:172.21.15.117:3300/0 2024-05-15T13:20:24.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.720+0000 7f0ad6dc6640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ad00ae1f0 con 0x7f0ad0070b80 2024-05-15T13:20:24.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.720+0000 7f0ad6dc6640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:24.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.720+0000 7f0ad6dc6640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ad006c150 con 0x7f0ad00ad820 2024-05-15T13:20:24.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.720+0000 7f0ad6dc6640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:24.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.720+0000 7f0ad6dc6640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad006fab0 con 0x7f0ad006bd70 2024-05-15T13:20:24.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.720+0000 7f0ad6dc6640 10 monclient(hunting): _renew_subs 2024-05-15T13:20:24.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.720+0000 7f0ad6dc6640 10 monclient(hunting): authenticate will time out at 3587.962562s 2024-05-15T13:20:24.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.721+0000 7f0ad4b62640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0ad006bd70 0x7f0ad0142940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:24.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.721+0000 7f0acffff640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f0ad0070b80 0x7f0ad0072570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:54058/0 (socket says 172.21.15.117:54058) 2024-05-15T13:20:24.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.721+0000 7f0acffff640 1 -- 172.21.15.117:0/2190869405 learned_addr learned my addr 172.21.15.117:0/2190869405 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-05-15T13:20:24.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.721+0000 7f0ad5363640 1 --2- 172.21.15.117:0/2190869405 >> v2:172.21.15.117:3300/0 conn(0x7f0ad00ad820 0x7f0ad00adcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:24.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.721+0000 7f0ad4b62640 10 monclient(hunting): get_auth_request con 0x7f0ad006bd70 auth_method 0 2024-05-15T13:20:24.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.721+0000 7f0ad4b62640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:24.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.721+0000 7f0ad4b62640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:24.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.721+0000 7f0ad4b62640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:24.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.722+0000 7f0ad5363640 10 monclient(hunting): get_auth_request con 0x7f0ad00ad820 auth_method 0 2024-05-15T13:20:24.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.722+0000 7f0ad5363640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:24.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.722+0000 7f0ad5363640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:24.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.722+0000 7f0ad5363640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:24.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.722+0000 7f0ad4b62640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:24.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.722+0000 7f0ad4b62640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:24.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.722+0000 7f0ad4b62640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:24.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.722+0000 7f0ad5363640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:24.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.722+0000 7f0ad5363640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:24.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.722+0000 7f0ad5363640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:24.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.723+0000 7f0ad4b62640 10 monclient(hunting): handle_auth_done global_id 18698 payload 274 2024-05-15T13:20:24.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.723+0000 7f0ad4b62640 1 -- 172.21.15.117:0/2190869405 >> v2:172.21.15.117:3300/0 conn(0x7f0ad00ad820 msgr2=0x7f0ad00adcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T13:20:24.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.723+0000 7f0ad4b62640 1 --2- 172.21.15.117:0/2190869405 >> v2:172.21.15.117:3300/0 conn(0x7f0ad00ad820 0x7f0ad00adcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:24.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.724+0000 7f0ad4b62640 1 -- 172.21.15.117:0/2190869405 >> v1:172.21.15.117:6789/0 conn(0x7f0ad0070b80 legacy=0x7f0ad0072570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:24.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.724+0000 7f0ad4b62640 10 monclient: _finish_hunting 0 2024-05-15T13:20:24.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.724+0000 7f0ad4b62640 1 monclient: found mon.noname-b 2024-05-15T13:20:24.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.724+0000 7f0ad4b62640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:20:24.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.724+0000 7f0ad4b62640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.117:3301/0 2024-05-15T13:20:24.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.724+0000 7f0ad4b62640 1 -- 172.21.15.117:0/2190869405 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ad0072c80 con 0x7f0ad006bd70 2024-05-15T13:20:24.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.725+0000 7f0ad4b62640 1 --2- 172.21.15.117:0/2190869405 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0ad006bd70 0x7f0ad0142940 secure :-1 s=READY pgs=3378 cs=0 l=1 rev1=1 crypto rx=0x7f0ac40097d0 tx=0x7f0ac402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68541b8832684092 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:24.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.729+0000 7f0acf7fe640 1 -- 172.21.15.117:0/2190869405 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac4035020 con 0x7f0ad006bd70 2024-05-15T13:20:24.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.729+0000 7f0acf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:24.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.729+0000 7f0acf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T13:20:24.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.729+0000 7f0acf7fe640 10 monclient: dump: 2024-05-15T13:20:24.728 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:24.728 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:24.728 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:24.728 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:24.728 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:24.728 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:24.728 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:24.729 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:24.729 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:24.729 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:24.729 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.729+0000 7f0acf7fe640 10 monclient: _finish_auth 0 2024-05-15T13:20:24.729 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.729+0000 7f0acf7fe640 10 monclient: _check_auth_tickets 2024-05-15T13:20:24.729 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.729+0000 7f0acf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:20:24.729 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.729+0000 7f0acf7fe640 1 -- 172.21.15.117:0/2190869405 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ac4031800 con 0x7f0ad006bd70 2024-05-15T13:20:24.729 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.729+0000 7f0acf7fe640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:20:24.729 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.729+0000 7f0acf7fe640 1 -- 172.21.15.117:0/2190869405 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac4031bd0 con 0x7f0ad006bd70 2024-05-15T13:20:24.729 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.729+0000 7f0ad6dc6640 5 monclient: authenticate success, global_id 18698 2024-05-15T13:20:24.729 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.729+0000 7f0ad6dc6640 10 monclient: get_monmap_and_config success 2024-05-15T13:20:24.729 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.729+0000 7f0ad6dc6640 4 set_mon_vals no callback set 2024-05-15T13:20:24.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.729+0000 7f0acf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:24.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.729+0000 7f0acf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T13:20:24.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.729+0000 7f0acf7fe640 10 monclient: dump: 2024-05-15T13:20:24.730 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:24.730 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:24.730 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:24.730 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:24.730 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:24.730 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:24.730 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:24.730 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:24.730 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:24.730 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:24.734 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.730+0000 7f0ad6dc6640 10 monclient: shutdown 2024-05-15T13:20:24.734 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.730+0000 7f0ad6dc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:20:24.734 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.730+0000 7f0ad6dc6640 1 -- 172.21.15.117:0/2190869405 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0ad006bd70 msgr2=0x7f0ad0142940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:24.734 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.730+0000 7f0ad6dc6640 1 --2- 172.21.15.117:0/2190869405 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0ad006bd70 0x7f0ad0142940 secure :-1 s=READY pgs=3378 cs=0 l=1 rev1=1 crypto rx=0x7f0ac40097d0 tx=0x7f0ac402b8a0 comp rx=0 tx=0).stop 2024-05-15T13:20:24.734 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.730+0000 7f0ad6dc6640 1 -- 172.21.15.117:0/2190869405 shutdown_connections 2024-05-15T13:20:24.734 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.730+0000 7f0ad6dc6640 1 --2- 172.21.15.117:0/2190869405 >> v2:172.21.15.117:3300/0 conn(0x7f0ad00ad820 0x7f0ad00adcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:24.735 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.730+0000 7f0ad6dc6640 1 --2- 172.21.15.117:0/2190869405 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0ad006bd70 0x7f0ad0142940 unknown :-1 s=CLOSED pgs=3378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:24.735 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.730+0000 7f0ad6dc6640 1 -- 172.21.15.117:0/2190869405 >> 172.21.15.117:0/2190869405 conn(0x7f0ad006ada0 msgr2=0x7f0ad006b1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:20:24.735 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.730+0000 7f0ad6dc6640 1 -- 172.21.15.117:0/2190869405 shutdown_connections 2024-05-15T13:20:24.735 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.730+0000 7f0ad6dc6640 1 -- 172.21.15.117:0/2190869405 wait complete. 2024-05-15T13:20:24.735 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.731+0000 7f0ad6dc6640 10 monclient: build_initial_monmap 2024-05-15T13:20:24.735 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.731+0000 7f0ad6dc6640 10 monclient: monmap: 2024-05-15T13:20:24.735 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T13:20:24.735 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T13:20:24.735 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-05-15T13:20:24.735 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-05-15T13:20:24.735 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T13:20:24.735 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:24.736 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T13:20:24.736 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T13:20:24.736 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T13:20:24.736 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:24.736 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.731+0000 7f0ad6dc6640 1 librados: starting msgr at 2024-05-15T13:20:24.736 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.731+0000 7f0ad6dc6640 1 librados: starting objecter 2024-05-15T13:20:24.736 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.731+0000 7f0ad6dc6640 1 Processor -- start 2024-05-15T13:20:24.736 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.731+0000 7f0ad6dc6640 1 -- start start 2024-05-15T13:20:24.736 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.731+0000 7f0ad6dc6640 1 librados: setting wanted keys 2024-05-15T13:20:24.736 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.731+0000 7f0ad6dc6640 1 librados: calling monclient init 2024-05-15T13:20:24.736 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.731+0000 7f0ad6dc6640 10 monclient: init 2024-05-15T13:20:24.737 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.732+0000 7f0ad6dc6640 10 monclient: _reopen_session rank -1 2024-05-15T13:20:24.737 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.732+0000 7f0ad6dc6640 10 monclient: _add_conns ranks=[0,2,1] 2024-05-15T13:20:24.737 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.732+0000 7f0ad6dc6640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ad006bd70 0x7f0ad0138340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:24.737 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.732+0000 7f0ad6dc6640 10 monclient(hunting): picked mon.noname-a con 0x7f0ad006bd70 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T13:20:24.737 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.732+0000 7f0ad6dc6640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0ad00ad820 0x7f0ad013acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:24.737 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.732+0000 7f0ad6dc6640 10 monclient(hunting): picked mon.noname-c con 0x7f0ad00ad820 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T13:20:24.737 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.732+0000 7f0ad6dc6640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0ad0139250 0x7f0ad013ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:24.737 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.732+0000 7f0ad6dc6640 10 monclient(hunting): picked mon.noname-b con 0x7f0ad0139250 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T13:20:24.737 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.732+0000 7f0ad6dc6640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:24.737 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.732+0000 7f0ad6dc6640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad006f6b0 con 0x7f0ad006bd70 2024-05-15T13:20:24.737 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.732+0000 7f0ad6dc6640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:24.737 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.732+0000 7f0ad6dc6640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad013cf40 con 0x7f0ad0139250 2024-05-15T13:20:24.737 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.732+0000 7f0ad6dc6640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:24.738 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.732+0000 7f0ad6dc6640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad0149cd0 con 0x7f0ad00ad820 2024-05-15T13:20:24.738 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.732+0000 7f0ad6dc6640 10 monclient(hunting): _renew_subs 2024-05-15T13:20:24.738 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.732+0000 7f0ad6dc6640 10 monclient(hunting): authenticate will time out at 3587.974236s 2024-05-15T13:20:24.738 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.732+0000 7f0acffff640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0ad00ad820 0x7f0ad013acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:24.738 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.732+0000 7f0ad4b62640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ad006bd70 0x7f0ad0138340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:24.738 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.733+0000 7f0ad5363640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0ad0139250 0x7f0ad013ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:24.738 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.733+0000 7f0ad4b62640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ad006bd70 0x7f0ad0138340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:54696/0 (socket says 172.21.15.117:54696) 2024-05-15T13:20:24.738 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.733+0000 7f0ad4b62640 1 -- 172.21.15.117:0/2305949113 learned_addr learned my addr 172.21.15.117:0/2305949113 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T13:20:24.739 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.733+0000 7f0ad4b62640 10 monclient(hunting): get_auth_request con 0x7f0ad006bd70 auth_method 0 2024-05-15T13:20:24.739 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.733+0000 7f0ad4b62640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:24.739 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.733+0000 7f0ad4b62640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:24.739 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.733+0000 7f0ad4b62640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:24.739 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.733+0000 7f0ad5363640 10 monclient(hunting): get_auth_request con 0x7f0ad0139250 auth_method 0 2024-05-15T13:20:24.739 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.733+0000 7f0ad5363640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:24.739 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.733+0000 7f0ad5363640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:24.739 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.733+0000 7f0ad5363640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:24.739 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.733+0000 7f0acffff640 10 monclient(hunting): get_auth_request con 0x7f0ad00ad820 auth_method 0 2024-05-15T13:20:24.739 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.733+0000 7f0acffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:24.739 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.733+0000 7f0acffff640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:24.739 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.733+0000 7f0acffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:24.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.733+0000 7f0ad4b62640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:24.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.733+0000 7f0ad4b62640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:24.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.733+0000 7f0ad4b62640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:24.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.733+0000 7f0ad5363640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:24.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.733+0000 7f0ad5363640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:24.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.733+0000 7f0ad5363640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:24.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.733+0000 7f0acffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:24.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.733+0000 7f0acffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:24.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.733+0000 7f0acffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:24.742 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.736+0000 7f0ad4b62640 10 monclient(hunting): handle_auth_done global_id 15975 payload 824 2024-05-15T13:20:24.742 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.736+0000 7f0ad4b62640 1 -- 172.21.15.117:0/2305949113 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0ad00ad820 msgr2=0x7f0ad013acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:24.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.736+0000 7f0ad4b62640 1 --2- 172.21.15.117:0/2305949113 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0ad00ad820 0x7f0ad013acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:24.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.736+0000 7f0ad4b62640 1 -- 172.21.15.117:0/2305949113 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0ad0139250 msgr2=0x7f0ad013ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:24.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.736+0000 7f0ad4b62640 1 --2- 172.21.15.117:0/2305949113 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0ad0139250 0x7f0ad013ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:24.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.736+0000 7f0ad4b62640 10 monclient: _finish_hunting 0 2024-05-15T13:20:24.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.736+0000 7f0ad4b62640 1 monclient: found mon.noname-a 2024-05-15T13:20:24.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.736+0000 7f0ad4b62640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:20:24.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.736+0000 7f0ad4b62640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.117:3300/0 2024-05-15T13:20:24.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.736+0000 7f0ad4b62640 1 -- 172.21.15.117:0/2305949113 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ad013b570 con 0x7f0ad006bd70 2024-05-15T13:20:24.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.739+0000 7f0ad4b62640 1 --2- 172.21.15.117:0/2305949113 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ad006bd70 0x7f0ad0138340 secure :-1 s=READY pgs=3287 cs=0 l=1 rev1=1 crypto rx=0x7f0ac402beb0 tx=0x7f0ac40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:24.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.739+0000 7f0acd7fa640 10 client.?.objecter ms_handle_connect 0x7f0ad006bd70 2024-05-15T13:20:24.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.739+0000 7f0acd7fa640 10 client.?.objecter resend_mon_ops 2024-05-15T13:20:24.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.743+0000 7f0acd7fa640 1 -- 172.21.15.117:0/2305949113 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac4035040 con 0x7f0ad006bd70 2024-05-15T13:20:24.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.743+0000 7f0acd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:24.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.743+0000 7f0acd7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T13:20:24.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.743+0000 7f0acd7fa640 10 monclient: dump: 2024-05-15T13:20:24.743 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:24.744 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:24.744 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:24.744 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:24.744 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:24.744 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:24.744 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:24.744 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:24.744 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:24.744 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:24.744 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.743+0000 7f0acd7fa640 10 monclient: _finish_auth 0 2024-05-15T13:20:24.744 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.743+0000 7f0acd7fa640 10 monclient: _check_auth_tickets 2024-05-15T13:20:24.744 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.743+0000 7f0acd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:20:24.744 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.743+0000 7f0acd7fa640 1 -- 172.21.15.117:0/2305949113 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ac403ad00 con 0x7f0ad006bd70 2024-05-15T13:20:24.744 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.743+0000 7f0ad6dc6640 5 monclient: authenticate success, global_id 15975 2024-05-15T13:20:24.744 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.743+0000 7f0ad6dc6640 10 monclient: _renew_subs 2024-05-15T13:20:24.744 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.743+0000 7f0ad6dc6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T13:20:24.745 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.743+0000 7f0ad6dc6640 1 -- 172.21.15.117:0/2305949113 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ad00a9cc0 con 0x7f0ad006bd70 2024-05-15T13:20:24.745 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.743+0000 7f0acd7fa640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:20:24.745 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.743+0000 7f0acd7fa640 1 -- 172.21.15.117:0/2305949113 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac40466a0 con 0x7f0ad006bd70 2024-05-15T13:20:24.745 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.743+0000 7f0acd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:24.745 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.743+0000 7f0acd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T13:20:24.745 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.743+0000 7f0acd7fa640 10 monclient: dump: 2024-05-15T13:20:24.745 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:24.745 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:24.745 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:24.745 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:24.745 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:24.745 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:24.745 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:24.745 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:24.745 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:24.746 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:24.746 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.743+0000 7f0aceffd640 4 set_mon_vals no callback set 2024-05-15T13:20:24.746 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.743+0000 7f0ad6dc6640 10 client.15975.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:20:24.746 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.743+0000 7f0ad6dc6640 10 monclient: _renew_subs 2024-05-15T13:20:24.746 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.743+0000 7f0ad6dc6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T13:20:24.746 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.743+0000 7f0ad6dc6640 1 -- 172.21.15.117:0/2305949113 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ad013bf90 con 0x7f0ad006bd70 2024-05-15T13:20:24.746 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.743+0000 7f0ad6dc6640 1 librados: init done 2024-05-15T13:20:24.746 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.744+0000 7f0ad6dc6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T13:20:24.746 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.744+0000 7f0ad6dc6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:20:24.746 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.744+0000 7f0ad6dc6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T13:20:24.746 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.744+0000 7f0ad6dc6640 1 -- 172.21.15.117:0/2305949113 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ad004f0e0 con 0x7f0ad006bd70 2024-05-15T13:20:24.760 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.762+0000 7f0acd7fa640 1 -- 172.21.15.117:0/2305949113 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f0ac4038020 con 0x7f0ad006bd70 2024-05-15T13:20:24.760 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.762+0000 7f0acd7fa640 1 --2- 172.21.15.117:0/2305949113 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f0ab406ceb0 0x7f0ab406f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:24.761 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.762+0000 7f0acffff640 1 --2- 172.21.15.117:0/2305949113 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f0ab406ceb0 0x7f0ab406f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:24.761 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.762+0000 7f0acffff640 10 monclient: get_auth_request con 0x7f0ab406ceb0 auth_method 0 2024-05-15T13:20:24.761 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.763+0000 7f0acd7fa640 10 client.15975.objecter ms_handle_connect 0x7f0ab406ceb0 2024-05-15T13:20:24.761 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.763+0000 7f0acffff640 1 --2- 172.21.15.117:0/2305949113 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f0ab406ceb0 0x7f0ab406f370 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f0ad01355a0 tx=0x7f0abc007e80 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:24.764 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.766+0000 7f0acd7fa640 1 -- 172.21.15.117:0/2305949113 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(1224..1224 src has 912..1224) v4 ==== 6365+0+0 (secure 0 0 0) 0x7f0ac4078db0 con 0x7f0ad006bd70 2024-05-15T13:20:24.764 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.766+0000 7f0acd7fa640 10 client.15975.objecter ms_dispatch 0x7f0ad0000b90 osd_map(1224..1224 src has 912..1224) v4 2024-05-15T13:20:24.764 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.766+0000 7f0acd7fa640 3 client.15975.objecter handle_osd_map got epochs [1224,1224] > 0 2024-05-15T13:20:24.764 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.766+0000 7f0acd7fa640 3 client.15975.objecter handle_osd_map decoding full epoch 1224 2024-05-15T13:20:24.764 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:24.766+0000 7f0acd7fa640 20 client.15975.objecter dump_active .. 0 homeless 2024-05-15T13:20:25.000 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.001+0000 7f0acd7fa640 1 -- 172.21.15.117:0/2305949113 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f0ad013b570 con 0x7f0ad006bd70 2024-05-15T13:20:25.000 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.001+0000 7f0acd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:20:25.000 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.001+0000 7f0acd7fa640 10 monclient: _finish_command 1 = system:0 2024-05-15T13:20:25.193 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.194+0000 7f0ad6dc6640 1 -- 172.21.15.117:0/2305949113 --> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0ad0149ab0 con 0x7f0ab406ceb0 2024-05-15T13:20:25.198 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.200+0000 7f0acd7fa640 1 -- 172.21.15.117:0/2305949113 <== mgr.4107 v2:172.21.15.117:6800/1785766523 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+116276 (secure 0 0 0) 0x7f0ad0149ab0 con 0x7f0ab406ceb0 2024-05-15T13:20:25.200 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T13:20:25.200 INFO:teuthology.orchestra.run.smithi117.stderr:dumped all 2024-05-15T13:20:25.202 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.202+0000 7f0ad6dc6640 10 librados: watch_flush enter 2024-05-15T13:20:25.202 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.202+0000 7f0ad6dc6640 10 librados: watch_flush exit 2024-05-15T13:20:25.202 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.202+0000 7f0ad6dc6640 20 client.15975.objecter shutdown clearing up homeless session... 2024-05-15T13:20:25.202 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.202+0000 7f0ad6dc6640 10 client.15975.objecter successfully canceled tick 2024-05-15T13:20:25.202 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.203+0000 7f0ad6dc6640 1 -- 172.21.15.117:0/2305949113 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f0ab406ceb0 msgr2=0x7f0ab406f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:25.202 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.203+0000 7f0ad6dc6640 1 --2- 172.21.15.117:0/2305949113 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f0ab406ceb0 0x7f0ab406f370 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f0ad01355a0 tx=0x7f0abc007e80 comp rx=0 tx=0).stop 2024-05-15T13:20:25.202 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.203+0000 7f0ad6dc6640 10 monclient: shutdown 2024-05-15T13:20:25.202 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.203+0000 7f0ad6dc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:20:25.202 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.203+0000 7f0ad6dc6640 1 -- 172.21.15.117:0/2305949113 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ad006bd70 msgr2=0x7f0ad0138340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:25.202 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.203+0000 7f0ad6dc6640 1 --2- 172.21.15.117:0/2305949113 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ad006bd70 0x7f0ad0138340 secure :-1 s=READY pgs=3287 cs=0 l=1 rev1=1 crypto rx=0x7f0ac402beb0 tx=0x7f0ac40046c0 comp rx=0 tx=0).stop 2024-05-15T13:20:25.202 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.203+0000 7f0ad6dc6640 1 -- 172.21.15.117:0/2305949113 shutdown_connections 2024-05-15T13:20:25.202 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.203+0000 7f0ad6dc6640 1 --2- 172.21.15.117:0/2305949113 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f0ab406ceb0 0x7f0ab406f370 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:25.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.203+0000 7f0ad6dc6640 1 --2- 172.21.15.117:0/2305949113 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0ad0139250 0x7f0ad013ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:25.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.203+0000 7f0ad6dc6640 1 --2- 172.21.15.117:0/2305949113 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0ad00ad820 0x7f0ad013acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:25.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.203+0000 7f0ad6dc6640 1 --2- 172.21.15.117:0/2305949113 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ad006bd70 0x7f0ad0138340 unknown :-1 s=CLOSED pgs=3287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:25.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.203+0000 7f0ad6dc6640 1 -- 172.21.15.117:0/2305949113 >> 172.21.15.117:0/2305949113 conn(0x7f0ad006ada0 msgr2=0x7f0ad00b2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:20:25.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.204+0000 7f0ad6dc6640 1 -- 172.21.15.117:0/2305949113 shutdown_connections 2024-05-15T13:20:25.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.204+0000 7f0ad6dc6640 1 -- 172.21.15.117:0/2305949113 wait complete. 2024-05-15T13:20:25.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.204+0000 7f0ad6dc6640 1 librados: shutdown 2024-05-15T13:20:25.218 INFO:teuthology.orchestra.run.smithi117.stdout:{"pg_ready":true,"pg_map":{"version":2355,"stamp":"2024-05-15T13:20:24.612592+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":85,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":602048,"kb_used_data":41680,"kb_used_omap":12,"kb_used_meta":560243,"kb_avail":754372672,"statfs":{"total":773094113280,"available":772477616128,"internally_reserved":0,"allocated":42680320,"data_stored":21281579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":573689394},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":141,"apply_latency_ms":141,"commit_latency_ns":141000000,"apply_latency_ns":141000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.009369"},"pg_stats":[{"pgid":"6.16","version":"0'0","reported_seq":2354,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.597416+0000","last_change":"2024-05-15T12:44:59.567389+0000","last_active":"2024-05-15T13:20:19.597416+0000","last_peered":"2024-05-15T13:20:19.597416+0000","last_clean":"2024-05-15T13:20:19.597416+0000","last_became_active":"2024-05-15T12:44:59.550282+0000","last_became_peered":"2024-05-15T12:44:59.550282+0000","last_unstale":"2024-05-15T13:20:19.597416+0000","last_undegraded":"2024-05-15T13:20:19.597416+0000","last_fullsized":"2024-05-15T13:20:19.597416+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:44:22.143842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":2369,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.612783+0000","last_change":"2024-05-15T12:44:51.243387+0000","last_active":"2024-05-15T13:20:19.612783+0000","last_peered":"2024-05-15T13:20:19.612783+0000","last_clean":"2024-05-15T13:20:19.612783+0000","last_became_active":"2024-05-15T12:44:51.202412+0000","last_became_peered":"2024-05-15T12:44:51.202412+0000","last_unstale":"2024-05-15T13:20:19.612783+0000","last_undegraded":"2024-05-15T13:20:19.612783+0000","last_fullsized":"2024-05-15T13:20:19.612783+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:46:23.223713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":2355,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.590005+0000","last_change":"2024-05-15T12:45:13.532350+0000","last_active":"2024-05-15T13:20:19.590005+0000","last_peered":"2024-05-15T13:20:19.590005+0000","last_clean":"2024-05-15T13:20:19.590005+0000","last_became_active":"2024-05-15T12:45:11.468881+0000","last_became_peered":"2024-05-15T12:45:11.468881+0000","last_unstale":"2024-05-15T13:20:19.590005+0000","last_undegraded":"2024-05-15T13:20:19.590005+0000","last_fullsized":"2024-05-15T13:20:19.590005+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T15:53:17.739425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":2357,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.512008+0000","last_change":"2024-05-15T12:44:55.797152+0000","last_active":"2024-05-15T13:20:19.512008+0000","last_peered":"2024-05-15T13:20:19.512008+0000","last_clean":"2024-05-15T13:20:19.512008+0000","last_became_active":"2024-05-15T12:44:55.775514+0000","last_became_peered":"2024-05-15T12:44:55.775514+0000","last_unstale":"2024-05-15T13:20:19.512008+0000","last_undegraded":"2024-05-15T13:20:19.512008+0000","last_fullsized":"2024-05-15T13:20:19.512008+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:09:17.199545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":2355,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.614535+0000","last_change":"2024-05-15T12:44:57.700224+0000","last_active":"2024-05-15T13:20:19.614535+0000","last_peered":"2024-05-15T13:20:19.614535+0000","last_clean":"2024-05-15T13:20:19.614535+0000","last_became_active":"2024-05-15T12:44:57.673614+0000","last_became_peered":"2024-05-15T12:44:57.673614+0000","last_unstale":"2024-05-15T13:20:19.614535+0000","last_undegraded":"2024-05-15T13:20:19.614535+0000","last_fullsized":"2024-05-15T13:20:19.614535+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-17T00:09:31.317495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":2355,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.583562+0000","last_change":"2024-05-15T12:44:59.486234+0000","last_active":"2024-05-15T13:20:19.583562+0000","last_peered":"2024-05-15T13:20:19.583562+0000","last_clean":"2024-05-15T13:20:19.583562+0000","last_became_active":"2024-05-15T12:44:59.480451+0000","last_became_peered":"2024-05-15T12:44:59.480451+0000","last_unstale":"2024-05-15T13:20:19.583562+0000","last_undegraded":"2024-05-15T13:20:19.583562+0000","last_fullsized":"2024-05-15T13:20:19.583562+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:28:22.234323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":2361,"reported_epoch":1221,"state":"active+clean","last_fresh":"2024-05-15T13:20:20.722753+0000","last_change":"2024-05-15T12:44:54.924153+0000","last_active":"2024-05-15T13:20:20.722753+0000","last_peered":"2024-05-15T13:20:20.722753+0000","last_clean":"2024-05-15T13:20:20.722753+0000","last_became_active":"2024-05-15T12:44:54.914910+0000","last_became_peered":"2024-05-15T12:44:54.914910+0000","last_unstale":"2024-05-15T13:20:20.722753+0000","last_undegraded":"2024-05-15T13:20:20.722753+0000","last_fullsized":"2024-05-15T13:20:20.722753+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-17T00:36:49.391069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":2356,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.564063+0000","last_change":"2024-05-15T12:44:59.650327+0000","last_active":"2024-05-15T13:20:19.564063+0000","last_peered":"2024-05-15T13:20:19.564063+0000","last_clean":"2024-05-15T13:20:19.564063+0000","last_became_active":"2024-05-15T12:44:59.595062+0000","last_became_peered":"2024-05-15T12:44:59.595062+0000","last_unstale":"2024-05-15T13:20:19.564063+0000","last_undegraded":"2024-05-15T13:20:19.564063+0000","last_fullsized":"2024-05-15T13:20:19.564063+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T12:52:31.383828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":2371,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.607147+0000","last_change":"2024-05-15T12:44:51.802252+0000","last_active":"2024-05-15T13:20:19.607147+0000","last_peered":"2024-05-15T13:20:19.607147+0000","last_clean":"2024-05-15T13:20:19.607147+0000","last_became_active":"2024-05-15T12:44:51.785998+0000","last_became_peered":"2024-05-15T12:44:51.785998+0000","last_unstale":"2024-05-15T13:20:19.607147+0000","last_undegraded":"2024-05-15T13:20:19.607147+0000","last_fullsized":"2024-05-15T13:20:19.607147+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:00:30.969147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":2372,"reported_epoch":1221,"state":"active+clean","last_fresh":"2024-05-15T13:20:20.670827+0000","last_change":"2024-05-15T12:44:53.614201+0000","last_active":"2024-05-15T13:20:20.670827+0000","last_peered":"2024-05-15T13:20:20.670827+0000","last_clean":"2024-05-15T13:20:20.670827+0000","last_became_active":"2024-05-15T12:44:53.608407+0000","last_became_peered":"2024-05-15T12:44:53.608407+0000","last_unstale":"2024-05-15T13:20:20.670827+0000","last_undegraded":"2024-05-15T13:20:20.670827+0000","last_fullsized":"2024-05-15T13:20:20.670827+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:45:19.480038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":2362,"reported_epoch":1222,"state":"active+clean","last_fresh":"2024-05-15T13:20:22.298535+0000","last_change":"2024-05-15T12:44:52.221672+0000","last_active":"2024-05-15T13:20:22.298535+0000","last_peered":"2024-05-15T13:20:22.298535+0000","last_clean":"2024-05-15T13:20:22.298535+0000","last_became_active":"2024-05-15T12:44:52.212480+0000","last_became_peered":"2024-05-15T12:44:52.212480+0000","last_unstale":"2024-05-15T13:20:22.298535+0000","last_undegraded":"2024-05-15T13:20:22.298535+0000","last_fullsized":"2024-05-15T13:20:22.298535+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:41:34.164739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":2371,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.609138+0000","last_change":"2024-05-15T12:44:56.446419+0000","last_active":"2024-05-15T13:20:19.609138+0000","last_peered":"2024-05-15T13:20:19.609138+0000","last_clean":"2024-05-15T13:20:19.609138+0000","last_became_active":"2024-05-15T12:44:56.197837+0000","last_became_peered":"2024-05-15T12:44:56.197837+0000","last_unstale":"2024-05-15T13:20:19.609138+0000","last_undegraded":"2024-05-15T13:20:19.609138+0000","last_fullsized":"2024-05-15T13:20:19.609138+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:33:22.475565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":2360,"reported_epoch":1221,"state":"active+clean","last_fresh":"2024-05-15T13:20:20.814121+0000","last_change":"2024-05-15T12:44:53.399230+0000","last_active":"2024-05-15T13:20:20.814121+0000","last_peered":"2024-05-15T13:20:20.814121+0000","last_clean":"2024-05-15T13:20:20.814121+0000","last_became_active":"2024-05-15T12:44:53.377773+0000","last_became_peered":"2024-05-15T12:44:53.377773+0000","last_unstale":"2024-05-15T13:20:20.814121+0000","last_undegraded":"2024-05-15T13:20:20.814121+0000","last_fullsized":"2024-05-15T13:20:20.814121+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:12:39.424987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'76","reported_seq":2393,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.589082+0000","last_change":"2024-05-15T12:43:04.763767+0000","last_active":"2024-05-15T13:20:19.589082+0000","last_peered":"2024-05-15T13:20:19.589082+0000","last_clean":"2024-05-15T13:20:19.589082+0000","last_became_active":"2024-05-15T12:43:04.148735+0000","last_became_peered":"2024-05-15T12:43:04.148735+0000","last_unstale":"2024-05-15T13:20:19.589082+0000","last_undegraded":"2024-05-15T13:20:19.589082+0000","last_fullsized":"2024-05-15T13:20:19.589082+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:42:53.071197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:42:53.071197+0000","last_clean_scrub_stamp":"2024-05-15T12:42:53.071197+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:45:23.691461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2370,"reported_epoch":1221,"state":"active+clean","last_fresh":"2024-05-15T13:20:20.778978+0000","last_change":"2024-05-15T12:43:09.453018+0000","last_active":"2024-05-15T13:20:20.778978+0000","last_peered":"2024-05-15T13:20:20.778978+0000","last_clean":"2024-05-15T13:20:20.778978+0000","last_became_active":"2024-05-15T12:43:09.445878+0000","last_became_peered":"2024-05-15T12:43:09.445878+0000","last_unstale":"2024-05-15T13:20:20.778978+0000","last_undegraded":"2024-05-15T13:20:20.778978+0000","last_fullsized":"2024-05-15T13:20:20.778978+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_clean_scrub_stamp":"2024-05-15T12:43:07.133352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:55:05.766298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":2364,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.629451+0000","last_change":"2024-05-15T12:44:50.705660+0000","last_active":"2024-05-15T13:20:19.629451+0000","last_peered":"2024-05-15T13:20:19.629451+0000","last_clean":"2024-05-15T13:20:19.629451+0000","last_became_active":"2024-05-15T12:44:50.696664+0000","last_became_peered":"2024-05-15T12:44:50.696664+0000","last_unstale":"2024-05-15T13:20:19.629451+0000","last_undegraded":"2024-05-15T13:20:19.629451+0000","last_fullsized":"2024-05-15T13:20:19.629451+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:15:31.819701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2371,"reported_epoch":1221,"state":"active+clean","last_fresh":"2024-05-15T13:20:20.720837+0000","last_change":"2024-05-15T12:43:08.704425+0000","last_active":"2024-05-15T13:20:20.720837+0000","last_peered":"2024-05-15T13:20:20.720837+0000","last_clean":"2024-05-15T13:20:20.720837+0000","last_became_active":"2024-05-15T12:43:08.677751+0000","last_became_peered":"2024-05-15T12:43:08.677751+0000","last_unstale":"2024-05-15T13:20:20.720837+0000","last_undegraded":"2024-05-15T13:20:20.720837+0000","last_fullsized":"2024-05-15T13:20:20.720837+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_clean_scrub_stamp":"2024-05-15T12:43:07.133352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-17T00:22:51.843061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":2355,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.638687+0000","last_change":"2024-05-15T12:44:57.579283+0000","last_active":"2024-05-15T13:20:19.638687+0000","last_peered":"2024-05-15T13:20:19.638687+0000","last_clean":"2024-05-15T13:20:19.638687+0000","last_became_active":"2024-05-15T12:44:57.552111+0000","last_became_peered":"2024-05-15T12:44:57.552111+0000","last_unstale":"2024-05-15T13:20:19.638687+0000","last_undegraded":"2024-05-15T13:20:19.638687+0000","last_fullsized":"2024-05-15T13:20:19.638687+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:23:14.388995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2372,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.618288+0000","last_change":"2024-05-15T12:43:08.634999+0000","last_active":"2024-05-15T13:20:19.618288+0000","last_peered":"2024-05-15T13:20:19.618288+0000","last_clean":"2024-05-15T13:20:19.618288+0000","last_became_active":"2024-05-15T12:43:08.600745+0000","last_became_peered":"2024-05-15T12:43:08.600745+0000","last_unstale":"2024-05-15T13:20:19.618288+0000","last_undegraded":"2024-05-15T13:20:19.618288+0000","last_fullsized":"2024-05-15T13:20:19.618288+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_clean_scrub_stamp":"2024-05-15T12:43:07.133352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:17:56.643544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":2373,"reported_epoch":1221,"state":"active+clean","last_fresh":"2024-05-15T13:20:20.656371+0000","last_change":"2024-05-15T12:44:51.361084+0000","last_active":"2024-05-15T13:20:20.656371+0000","last_peered":"2024-05-15T13:20:20.656371+0000","last_clean":"2024-05-15T13:20:20.656371+0000","last_became_active":"2024-05-15T12:44:51.354879+0000","last_became_peered":"2024-05-15T12:44:51.354879+0000","last_unstale":"2024-05-15T13:20:20.656371+0000","last_undegraded":"2024-05-15T13:20:20.656371+0000","last_fullsized":"2024-05-15T13:20:20.656371+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:53:33.707185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2370,"reported_epoch":1221,"state":"active+clean","last_fresh":"2024-05-15T13:20:20.765250+0000","last_change":"2024-05-15T12:43:09.375465+0000","last_active":"2024-05-15T13:20:20.765250+0000","last_peered":"2024-05-15T13:20:20.765250+0000","last_clean":"2024-05-15T13:20:20.765250+0000","last_became_active":"2024-05-15T12:43:09.363876+0000","last_became_peered":"2024-05-15T12:43:09.363876+0000","last_unstale":"2024-05-15T13:20:20.765250+0000","last_undegraded":"2024-05-15T13:20:20.765250+0000","last_fullsized":"2024-05-15T13:20:20.765250+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_clean_scrub_stamp":"2024-05-15T12:43:07.133352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:27:46.599385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":2364,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.594769+0000","last_change":"2024-05-15T12:44:54.188435+0000","last_active":"2024-05-15T13:20:19.594769+0000","last_peered":"2024-05-15T13:20:19.594769+0000","last_clean":"2024-05-15T13:20:19.594769+0000","last_became_active":"2024-05-15T12:44:54.179460+0000","last_became_peered":"2024-05-15T12:44:54.179460+0000","last_unstale":"2024-05-15T13:20:19.594769+0000","last_undegraded":"2024-05-15T13:20:19.594769+0000","last_fullsized":"2024-05-15T13:20:19.594769+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T12:52:27.875820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2382,"reported_epoch":1221,"state":"active+clean","last_fresh":"2024-05-15T13:20:20.653740+0000","last_change":"2024-05-15T12:43:09.384765+0000","last_active":"2024-05-15T13:20:20.653740+0000","last_peered":"2024-05-15T13:20:20.653740+0000","last_clean":"2024-05-15T13:20:20.653740+0000","last_became_active":"2024-05-15T12:43:09.377725+0000","last_became_peered":"2024-05-15T12:43:09.377725+0000","last_unstale":"2024-05-15T13:20:20.653740+0000","last_undegraded":"2024-05-15T13:20:20.653740+0000","last_fullsized":"2024-05-15T13:20:20.653740+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_clean_scrub_stamp":"2024-05-15T12:43:07.133352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:57:59.671602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"47'7","reported_seq":2377,"reported_epoch":1222,"state":"active+clean","last_fresh":"2024-05-15T13:20:22.279258+0000","last_change":"2024-05-15T12:44:58.436179+0000","last_active":"2024-05-15T13:20:22.279258+0000","last_peered":"2024-05-15T13:20:22.279258+0000","last_clean":"2024-05-15T13:20:22.279258+0000","last_became_active":"2024-05-15T12:44:58.318306+0000","last_became_peered":"2024-05-15T12:44:58.318306+0000","last_unstale":"2024-05-15T13:20:22.279258+0000","last_undegraded":"2024-05-15T13:20:22.279258+0000","last_fullsized":"2024-05-15T13:20:22.279258+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:59:46.310559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2370,"reported_epoch":1221,"state":"active+clean","last_fresh":"2024-05-15T13:20:20.794871+0000","last_change":"2024-05-15T12:43:08.778833+0000","last_active":"2024-05-15T13:20:20.794871+0000","last_peered":"2024-05-15T13:20:20.794871+0000","last_clean":"2024-05-15T13:20:20.794871+0000","last_became_active":"2024-05-15T12:43:08.773614+0000","last_became_peered":"2024-05-15T12:43:08.773614+0000","last_unstale":"2024-05-15T13:20:20.794871+0000","last_undegraded":"2024-05-15T13:20:20.794871+0000","last_fullsized":"2024-05-15T13:20:20.794871+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_clean_scrub_stamp":"2024-05-15T12:43:07.133352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:22:59.785687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":2361,"reported_epoch":1221,"state":"active+clean","last_fresh":"2024-05-15T13:20:20.771718+0000","last_change":"2024-05-15T12:45:08.669407+0000","last_active":"2024-05-15T13:20:20.771718+0000","last_peered":"2024-05-15T13:20:20.771718+0000","last_clean":"2024-05-15T13:20:20.771718+0000","last_became_active":"2024-05-15T12:45:08.661140+0000","last_became_peered":"2024-05-15T12:45:08.661140+0000","last_unstale":"2024-05-15T13:20:20.771718+0000","last_undegraded":"2024-05-15T13:20:20.771718+0000","last_fullsized":"2024-05-15T13:20:20.771718+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:55:43.700115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2366,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.599275+0000","last_change":"2024-05-15T12:43:09.478046+0000","last_active":"2024-05-15T13:20:19.599275+0000","last_peered":"2024-05-15T13:20:19.599275+0000","last_clean":"2024-05-15T13:20:19.599275+0000","last_became_active":"2024-05-15T12:43:09.448237+0000","last_became_peered":"2024-05-15T12:43:09.448237+0000","last_unstale":"2024-05-15T13:20:19.599275+0000","last_undegraded":"2024-05-15T13:20:19.599275+0000","last_fullsized":"2024-05-15T13:20:19.599275+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_clean_scrub_stamp":"2024-05-15T12:43:07.133352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:11:17.055288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":2355,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.612460+0000","last_change":"2024-05-15T12:45:18.657708+0000","last_active":"2024-05-15T13:20:19.612460+0000","last_peered":"2024-05-15T13:20:19.612460+0000","last_clean":"2024-05-15T13:20:19.612460+0000","last_became_active":"2024-05-15T12:45:18.642568+0000","last_became_peered":"2024-05-15T12:45:18.642568+0000","last_unstale":"2024-05-15T13:20:19.612460+0000","last_undegraded":"2024-05-15T13:20:19.612460+0000","last_fullsized":"2024-05-15T13:20:19.612460+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:21:30.095711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2369,"reported_epoch":1221,"state":"active+clean","last_fresh":"2024-05-15T13:20:20.815988+0000","last_change":"2024-05-15T12:43:09.429828+0000","last_active":"2024-05-15T13:20:20.815988+0000","last_peered":"2024-05-15T13:20:20.815988+0000","last_clean":"2024-05-15T13:20:20.815988+0000","last_became_active":"2024-05-15T12:43:09.424460+0000","last_became_peered":"2024-05-15T12:43:09.424460+0000","last_unstale":"2024-05-15T13:20:20.815988+0000","last_undegraded":"2024-05-15T13:20:20.815988+0000","last_fullsized":"2024-05-15T13:20:20.815988+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_clean_scrub_stamp":"2024-05-15T12:43:07.133352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:12:21.424590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":2362,"reported_epoch":1222,"state":"active+clean","last_fresh":"2024-05-15T13:20:22.277210+0000","last_change":"2024-05-15T12:44:52.129096+0000","last_active":"2024-05-15T13:20:22.277210+0000","last_peered":"2024-05-15T13:20:22.277210+0000","last_clean":"2024-05-15T13:20:22.277210+0000","last_became_active":"2024-05-15T12:44:52.088745+0000","last_became_peered":"2024-05-15T12:44:52.088745+0000","last_unstale":"2024-05-15T13:20:22.277210+0000","last_undegraded":"2024-05-15T13:20:22.277210+0000","last_fullsized":"2024-05-15T13:20:22.277210+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:16:34.966002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":2355,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.552600+0000","last_change":"2024-05-15T12:44:56.480904+0000","last_active":"2024-05-15T13:20:19.552600+0000","last_peered":"2024-05-15T13:20:19.552600+0000","last_clean":"2024-05-15T13:20:19.552600+0000","last_became_active":"2024-05-15T12:44:56.470246+0000","last_became_peered":"2024-05-15T12:44:56.470246+0000","last_unstale":"2024-05-15T13:20:19.552600+0000","last_undegraded":"2024-05-15T13:20:19.552600+0000","last_fullsized":"2024-05-15T13:20:19.552600+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:53:59.488101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":2361,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.610996+0000","last_change":"2024-05-15T12:44:51.677390+0000","last_active":"2024-05-15T13:20:19.610996+0000","last_peered":"2024-05-15T13:20:19.610996+0000","last_clean":"2024-05-15T13:20:19.610996+0000","last_became_active":"2024-05-15T12:44:51.660527+0000","last_became_peered":"2024-05-15T12:44:51.660527+0000","last_unstale":"2024-05-15T13:20:19.610996+0000","last_undegraded":"2024-05-15T13:20:19.610996+0000","last_fullsized":"2024-05-15T13:20:19.610996+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:25:49.920807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":2354,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.616476+0000","last_change":"2024-05-15T12:45:12.010393+0000","last_active":"2024-05-15T13:20:19.616476+0000","last_peered":"2024-05-15T13:20:19.616476+0000","last_clean":"2024-05-15T13:20:19.616476+0000","last_became_active":"2024-05-15T12:45:11.998256+0000","last_became_peered":"2024-05-15T12:45:11.998256+0000","last_unstale":"2024-05-15T13:20:19.616476+0000","last_undegraded":"2024-05-15T13:20:19.616476+0000","last_fullsized":"2024-05-15T13:20:19.616476+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:13:05.556724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":2360,"reported_epoch":1222,"state":"active+clean","last_fresh":"2024-05-15T13:20:22.317768+0000","last_change":"2024-05-15T12:44:55.564951+0000","last_active":"2024-05-15T13:20:22.317768+0000","last_peered":"2024-05-15T13:20:22.317768+0000","last_clean":"2024-05-15T13:20:22.317768+0000","last_became_active":"2024-05-15T12:44:55.557214+0000","last_became_peered":"2024-05-15T12:44:55.557214+0000","last_unstale":"2024-05-15T13:20:22.317768+0000","last_undegraded":"2024-05-15T13:20:22.317768+0000","last_fullsized":"2024-05-15T13:20:22.317768+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:53:15.834038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":2359,"reported_epoch":1221,"state":"active+clean","last_fresh":"2024-05-15T13:20:20.785961+0000","last_change":"2024-05-15T12:44:56.078343+0000","last_active":"2024-05-15T13:20:20.785961+0000","last_peered":"2024-05-15T13:20:20.785961+0000","last_clean":"2024-05-15T13:20:20.785961+0000","last_became_active":"2024-05-15T12:44:56.072618+0000","last_became_peered":"2024-05-15T12:44:56.072618+0000","last_unstale":"2024-05-15T13:20:20.785961+0000","last_undegraded":"2024-05-15T13:20:20.785961+0000","last_fullsized":"2024-05-15T13:20:20.785961+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:22:00.980595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":2364,"reported_epoch":1222,"state":"active+clean","last_fresh":"2024-05-15T13:20:22.332975+0000","last_change":"2024-05-15T12:44:55.746501+0000","last_active":"2024-05-15T13:20:22.332975+0000","last_peered":"2024-05-15T13:20:22.332975+0000","last_clean":"2024-05-15T13:20:22.332975+0000","last_became_active":"2024-05-15T12:44:55.739811+0000","last_became_peered":"2024-05-15T12:44:55.739811+0000","last_unstale":"2024-05-15T13:20:22.332975+0000","last_undegraded":"2024-05-15T13:20:22.332975+0000","last_fullsized":"2024-05-15T13:20:22.332975+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T12:49:40.051745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":2361,"reported_epoch":1221,"state":"active+clean","last_fresh":"2024-05-15T13:20:20.800642+0000","last_change":"2024-05-15T12:44:56.495925+0000","last_active":"2024-05-15T13:20:20.800642+0000","last_peered":"2024-05-15T13:20:20.800642+0000","last_clean":"2024-05-15T13:20:20.800642+0000","last_became_active":"2024-05-15T12:44:56.429996+0000","last_became_peered":"2024-05-15T12:44:56.429996+0000","last_unstale":"2024-05-15T13:20:20.800642+0000","last_undegraded":"2024-05-15T13:20:20.800642+0000","last_fullsized":"2024-05-15T13:20:20.800642+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:18:31.842665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":2353,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.604742+0000","last_change":"2024-05-15T12:45:18.556824+0000","last_active":"2024-05-15T13:20:19.604742+0000","last_peered":"2024-05-15T13:20:19.604742+0000","last_clean":"2024-05-15T13:20:19.604742+0000","last_became_active":"2024-05-15T12:45:18.546529+0000","last_became_peered":"2024-05-15T12:45:18.546529+0000","last_unstale":"2024-05-15T13:20:19.604742+0000","last_undegraded":"2024-05-15T13:20:19.604742+0000","last_fullsized":"2024-05-15T13:20:19.604742+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:07:19.212044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":2354,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.555166+0000","last_change":"2024-05-15T12:44:56.642476+0000","last_active":"2024-05-15T13:20:19.555166+0000","last_peered":"2024-05-15T13:20:19.555166+0000","last_clean":"2024-05-15T13:20:19.555166+0000","last_became_active":"2024-05-15T12:44:56.634257+0000","last_became_peered":"2024-05-15T12:44:56.634257+0000","last_unstale":"2024-05-15T13:20:19.555166+0000","last_undegraded":"2024-05-15T13:20:19.555166+0000","last_fullsized":"2024-05-15T13:20:19.555166+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T15:44:34.817095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":2361,"reported_epoch":1222,"state":"active+clean","last_fresh":"2024-05-15T13:20:22.287049+0000","last_change":"2024-05-15T12:44:52.504602+0000","last_active":"2024-05-15T13:20:22.287049+0000","last_peered":"2024-05-15T13:20:22.287049+0000","last_clean":"2024-05-15T13:20:22.287049+0000","last_became_active":"2024-05-15T12:44:52.483168+0000","last_became_peered":"2024-05-15T12:44:52.483168+0000","last_unstale":"2024-05-15T13:20:22.287049+0000","last_undegraded":"2024-05-15T13:20:22.287049+0000","last_fullsized":"2024-05-15T13:20:22.287049+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:17:24.835749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":2359,"reported_epoch":1221,"state":"active+clean","last_fresh":"2024-05-15T13:20:20.768531+0000","last_change":"2024-05-15T12:45:12.195717+0000","last_active":"2024-05-15T13:20:20.768531+0000","last_peered":"2024-05-15T13:20:20.768531+0000","last_clean":"2024-05-15T13:20:20.768531+0000","last_became_active":"2024-05-15T12:45:12.190202+0000","last_became_peered":"2024-05-15T12:45:12.190202+0000","last_unstale":"2024-05-15T13:20:20.768531+0000","last_undegraded":"2024-05-15T13:20:20.768531+0000","last_fullsized":"2024-05-15T13:20:20.768531+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:05:34.894666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":20,"seq":85899346373,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72784,"kb_used_data":5504,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299056,"statfs":{"total":96636764160,"available":96562233344,"internally_reserved":0,"allocated":5636096,"data_stored":2959141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[]},{"osd":0,"up_from":20,"seq":85899346372,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68220,"kb_used_data":5100,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303620,"statfs":{"total":96636764160,"available":96566906880,"internally_reserved":0,"allocated":5222400,"data_stored":2553456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[]},{"osd":2,"up_from":20,"seq":85899346372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94532,"kb_used_data":5112,"kb_used_omap":1,"kb_used_meta":89406,"kb_avail":94277308,"statfs":{"total":96636764160,"available":96539963392,"internally_reserved":0,"allocated":5234688,"data_stored":2561301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":91552198},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":6,"up_from":19,"seq":81604379076,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72404,"kb_used_data":5128,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299436,"statfs":{"total":96636764160,"available":96562622464,"internally_reserved":0,"allocated":5251072,"data_stored":2568766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[]},{"osd":4,"up_from":19,"seq":81604379076,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72376,"kb_used_data":5100,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299464,"statfs":{"total":96636764160,"available":96562651136,"internally_reserved":0,"allocated":5222400,"data_stored":2553456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509890,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72396,"kb_used_data":5116,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299444,"statfs":{"total":96636764160,"available":96562630656,"internally_reserved":0,"allocated":5238784,"data_stored":2561306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542595,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72796,"kb_used_data":5520,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299044,"statfs":{"total":96636764160,"available":96562221056,"internally_reserved":0,"allocated":5652480,"data_stored":2970697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[]},{"osd":7,"up_from":13,"seq":55834575300,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76540,"kb_used_data":5100,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94295300,"statfs":{"total":96636764160,"available":96558387200,"internally_reserved":0,"allocated":5222400,"data_stored":2553456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T13:20:25.220 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-15T13:20:25.322 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.323+0000 7ff79e1ba640 10 monclient: get_monmap_and_config 2024-05-15T13:20:25.322 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.323+0000 7ff79e1ba640 10 monclient: build_initial_monmap 2024-05-15T13:20:25.322 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.323+0000 7ff79e1ba640 10 monclient: monmap: 2024-05-15T13:20:25.322 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T13:20:25.322 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:25.322 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T13:20:25.324619+0000 2024-05-15T13:20:25.322 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T13:20:25.324619+0000 2024-05-15T13:20:25.323 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T13:20:25.323 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:25.323 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T13:20:25.323 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T13:20:25.323 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T13:20:25.323 INFO:teuthology.orchestra.run.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T13:20:25.323 INFO:teuthology.orchestra.run.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T13:20:25.323 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:25.325 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.324+0000 7ff79e1ba640 1 Processor -- start 2024-05-15T13:20:25.325 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.324+0000 7ff79e1ba640 1 -- start start 2024-05-15T13:20:25.325 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.324+0000 7ff79e1ba640 10 monclient: init 2024-05-15T13:20:25.325 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.324+0000 7ff79e1ba640 10 monclient: _reopen_session rank -1 2024-05-15T13:20:25.325 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.324+0000 7ff79e1ba640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-05-15T13:20:25.325 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.324+0000 7ff79e1ba640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7ff798066cc0 0x7ff7980ae170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:25.326 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.324+0000 7ff79e1ba640 10 monclient(hunting): picked mon.noname-a con 0x7ff798066cc0 addr v2:172.21.15.117:3300/0 2024-05-15T13:20:25.326 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.324+0000 7ff79e1ba640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7ff79813c230 0x7ff79813c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:25.326 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.324+0000 7ff79e1ba640 10 monclient(hunting): picked mon.noname-c con 0x7ff79813c230 addr v2:172.21.15.178:3300/0 2024-05-15T13:20:25.326 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.324+0000 7ff79e1ba640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff798146100 addr v1:172.21.15.117:6789/0 2024-05-15T13:20:25.326 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.324+0000 7ff79e1ba640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7980670a0 con 0x7ff798146100 2024-05-15T13:20:25.326 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.324+0000 7ff79e1ba640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:25.326 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.325+0000 7ff79e1ba640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff798145c70 con 0x7ff798066cc0 2024-05-15T13:20:25.326 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.325+0000 7ff79e1ba640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:25.326 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.325+0000 7ff79e1ba640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7981426f0 con 0x7ff79813c230 2024-05-15T13:20:25.326 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.325+0000 7ff79e1ba640 10 monclient(hunting): _renew_subs 2024-05-15T13:20:25.326 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.325+0000 7ff79e1ba640 10 monclient(hunting): authenticate will time out at 3588.566812s 2024-05-15T13:20:25.326 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.325+0000 7ff7977fe640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7ff798066cc0 0x7ff7980ae170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:25.326 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.325+0000 7ff796ffd640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7ff79813c230 0x7ff79813c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:25.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.325+0000 7ff797fff640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7ff798146100 0x7ff798146510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:54070/0 (socket says 172.21.15.117:54070) 2024-05-15T13:20:25.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.325+0000 7ff797fff640 1 -- 172.21.15.117:0/625827780 learned_addr learned my addr 172.21.15.117:0/625827780 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-05-15T13:20:25.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.325+0000 7ff796ffd640 10 monclient(hunting): get_auth_request con 0x7ff79813c230 auth_method 0 2024-05-15T13:20:25.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.325+0000 7ff796ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:25.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.325+0000 7ff796ffd640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:25.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.325+0000 7ff796ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:25.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.326+0000 7ff7977fe640 10 monclient(hunting): get_auth_request con 0x7ff798066cc0 auth_method 0 2024-05-15T13:20:25.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.326+0000 7ff7977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:25.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.326+0000 7ff7977fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:25.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.326+0000 7ff7977fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:25.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.326+0000 7ff796ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:25.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.326+0000 7ff796ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:25.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.326+0000 7ff796ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:25.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.326+0000 7ff7977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:25.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.326+0000 7ff7977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:25.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.326+0000 7ff7977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:25.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.328+0000 7ff796ffd640 10 monclient(hunting): handle_auth_done global_id 22120 payload 274 2024-05-15T13:20:25.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.328+0000 7ff796ffd640 1 -- 172.21.15.117:0/625827780 >> v2:172.21.15.117:3300/0 conn(0x7ff798066cc0 msgr2=0x7ff7980ae170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T13:20:25.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.328+0000 7ff796ffd640 1 --2- 172.21.15.117:0/625827780 >> v2:172.21.15.117:3300/0 conn(0x7ff798066cc0 0x7ff7980ae170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:25.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.328+0000 7ff796ffd640 1 -- 172.21.15.117:0/625827780 >> v1:172.21.15.117:6789/0 conn(0x7ff798146100 legacy=0x7ff798146510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:25.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.328+0000 7ff796ffd640 10 monclient: _finish_hunting 0 2024-05-15T13:20:25.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.328+0000 7ff796ffd640 1 monclient: found mon.noname-c 2024-05-15T13:20:25.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.328+0000 7ff796ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:20:25.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.328+0000 7ff796ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.178:3300/0 2024-05-15T13:20:25.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.328+0000 7ff796ffd640 1 -- 172.21.15.117:0/625827780 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff79806ffa0 con 0x7ff79813c230 2024-05-15T13:20:25.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.330+0000 7ff796ffd640 1 --2- 172.21.15.117:0/625827780 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff79813c230 0x7ff79813c6c0 secure :-1 s=READY pgs=5436 cs=0 l=1 rev1=1 crypto rx=0x7ff7980aed00 tx=0x7ff78002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee0f8d5ac5d7f1fc server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:25.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.333+0000 7ff7967fc640 1 -- 172.21.15.117:0/625827780 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7800315a0 con 0x7ff79813c230 2024-05-15T13:20:25.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.333+0000 7ff7967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:25.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.333+0000 7ff7967fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T13:20:25.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.333+0000 7ff7967fc640 10 monclient: dump: 2024-05-15T13:20:25.333 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:25.333 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:25.333 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:25.333 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:25.333 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:25.333 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:25.333 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:25.333 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:25.333 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:25.333 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:25.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.333+0000 7ff7967fc640 10 monclient: _finish_auth 0 2024-05-15T13:20:25.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.333+0000 7ff7967fc640 10 monclient: _check_auth_tickets 2024-05-15T13:20:25.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.333+0000 7ff7967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:20:25.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.333+0000 7ff7967fc640 1 -- 172.21.15.117:0/625827780 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff780037070 con 0x7ff79813c230 2024-05-15T13:20:25.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.333+0000 7ff79e1ba640 5 monclient: authenticate success, global_id 22120 2024-05-15T13:20:25.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.333+0000 7ff79e1ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T13:20:25.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.333+0000 7ff7967fc640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:20:25.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.333+0000 7ff7967fc640 1 -- 172.21.15.117:0/625827780 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff780031960 con 0x7ff79813c230 2024-05-15T13:20:25.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.333+0000 7ff7967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:25.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.334+0000 7ff7967fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T13:20:25.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.334+0000 7ff7967fc640 10 monclient: dump: 2024-05-15T13:20:25.334 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:25.334 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:25.334 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:25.334 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:25.334 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:25.335 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:25.335 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:25.335 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:25.335 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:25.335 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:25.335 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.334+0000 7ff79e1ba640 10 monclient: get_monmap_and_config success 2024-05-15T13:20:25.335 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.334+0000 7ff79e1ba640 4 set_mon_vals no callback set 2024-05-15T13:20:25.339 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.334+0000 7ff79e1ba640 10 monclient: shutdown 2024-05-15T13:20:25.339 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.334+0000 7ff79e1ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:20:25.339 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.334+0000 7ff79e1ba640 1 -- 172.21.15.117:0/625827780 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff79813c230 msgr2=0x7ff79813c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:25.339 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.334+0000 7ff79e1ba640 1 --2- 172.21.15.117:0/625827780 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff79813c230 0x7ff79813c6c0 secure :-1 s=READY pgs=5436 cs=0 l=1 rev1=1 crypto rx=0x7ff7980aed00 tx=0x7ff78002b840 comp rx=0 tx=0).stop 2024-05-15T13:20:25.339 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.334+0000 7ff79e1ba640 1 -- 172.21.15.117:0/625827780 shutdown_connections 2024-05-15T13:20:25.339 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.334+0000 7ff79e1ba640 1 --2- 172.21.15.117:0/625827780 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff79813c230 0x7ff79813c6c0 unknown :-1 s=CLOSED pgs=5436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:25.339 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.334+0000 7ff79e1ba640 1 --2- 172.21.15.117:0/625827780 >> v2:172.21.15.117:3300/0 conn(0x7ff798066cc0 0x7ff7980ae170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:25.339 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.334+0000 7ff79e1ba640 1 -- 172.21.15.117:0/625827780 >> 172.21.15.117:0/625827780 conn(0x7ff7980b4a60 msgr2=0x7ff7980b0f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:20:25.339 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.334+0000 7ff79e1ba640 1 -- 172.21.15.117:0/625827780 shutdown_connections 2024-05-15T13:20:25.339 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.334+0000 7ff79e1ba640 1 -- 172.21.15.117:0/625827780 wait complete. 2024-05-15T13:20:25.339 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.335+0000 7ff79e1ba640 10 monclient: build_initial_monmap 2024-05-15T13:20:25.339 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.335+0000 7ff79e1ba640 10 monclient: monmap: 2024-05-15T13:20:25.339 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T13:20:25.340 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T13:20:25.340 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-05-15T13:20:25.340 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-05-15T13:20:25.340 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T13:20:25.340 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:25.340 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T13:20:25.340 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T13:20:25.340 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T13:20:25.340 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:25.340 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.335+0000 7ff79e1ba640 1 librados: starting msgr at 2024-05-15T13:20:25.340 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.335+0000 7ff79e1ba640 1 librados: starting objecter 2024-05-15T13:20:25.340 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.335+0000 7ff79e1ba640 1 Processor -- start 2024-05-15T13:20:25.340 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.336+0000 7ff79e1ba640 1 -- start start 2024-05-15T13:20:25.340 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.336+0000 7ff79e1ba640 1 librados: setting wanted keys 2024-05-15T13:20:25.340 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.336+0000 7ff79e1ba640 1 librados: calling monclient init 2024-05-15T13:20:25.341 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.336+0000 7ff79e1ba640 10 monclient: init 2024-05-15T13:20:25.341 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.336+0000 7ff79e1ba640 10 monclient: _reopen_session rank -1 2024-05-15T13:20:25.341 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.336+0000 7ff79e1ba640 10 monclient: _add_conns ranks=[1,2,0] 2024-05-15T13:20:25.341 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.336+0000 7ff79e1ba640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff798066cc0 0x7ff7981cc770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:25.341 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.336+0000 7ff79e1ba640 10 monclient(hunting): picked mon.noname-b con 0x7ff798066cc0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T13:20:25.341 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.336+0000 7ff79e1ba640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff79813c230 0x7ff798135610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:25.341 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.336+0000 7ff79e1ba640 10 monclient(hunting): picked mon.noname-c con 0x7ff79813c230 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T13:20:25.341 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.336+0000 7ff79e1ba640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff798146100 0x7ff798135b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:25.341 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.336+0000 7ff79e1ba640 10 monclient(hunting): picked mon.noname-a con 0x7ff798146100 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T13:20:25.341 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.336+0000 7ff79e1ba640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:25.341 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.336+0000 7ff79e1ba640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff798072660 con 0x7ff798146100 2024-05-15T13:20:25.341 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.336+0000 7ff79e1ba640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:25.341 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.336+0000 7ff79e1ba640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7980724e0 con 0x7ff798066cc0 2024-05-15T13:20:25.341 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.336+0000 7ff79e1ba640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:25.341 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.336+0000 7ff79e1ba640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7980727e0 con 0x7ff79813c230 2024-05-15T13:20:25.341 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.336+0000 7ff79e1ba640 10 monclient(hunting): _renew_subs 2024-05-15T13:20:25.341 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.336+0000 7ff79e1ba640 10 monclient(hunting): authenticate will time out at 3588.578379s 2024-05-15T13:20:25.342 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.337+0000 7ff797fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff798146100 0x7ff798135b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:25.342 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.337+0000 7ff7977fe640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff798066cc0 0x7ff7981cc770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:25.342 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.337+0000 7ff796ffd640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff79813c230 0x7ff798135610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:25.342 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.337+0000 7ff7977fe640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff798066cc0 0x7ff7981cc770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:35986/0 (socket says 172.21.15.117:35986) 2024-05-15T13:20:25.342 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.337+0000 7ff7977fe640 1 -- 172.21.15.117:0/3890962144 learned_addr learned my addr 172.21.15.117:0/3890962144 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T13:20:25.342 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.337+0000 7ff7977fe640 10 monclient(hunting): get_auth_request con 0x7ff798066cc0 auth_method 0 2024-05-15T13:20:25.342 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.337+0000 7ff7977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:25.342 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.337+0000 7ff7977fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:25.342 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.337+0000 7ff7977fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:25.342 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.337+0000 7ff797fff640 10 monclient(hunting): get_auth_request con 0x7ff798146100 auth_method 0 2024-05-15T13:20:25.342 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.337+0000 7ff797fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:25.342 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.337+0000 7ff797fff640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:25.342 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.337+0000 7ff797fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:25.342 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.337+0000 7ff796ffd640 10 monclient(hunting): get_auth_request con 0x7ff79813c230 auth_method 0 2024-05-15T13:20:25.342 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.337+0000 7ff796ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:25.342 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.337+0000 7ff796ffd640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:25.343 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.337+0000 7ff796ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:25.343 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.337+0000 7ff7977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:25.343 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.337+0000 7ff7977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:25.343 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.337+0000 7ff7977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:25.343 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.337+0000 7ff797fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:25.343 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.337+0000 7ff797fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:25.343 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.337+0000 7ff797fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:25.343 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.338+0000 7ff796ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:25.343 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.338+0000 7ff796ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:25.343 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.338+0000 7ff796ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:25.343 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.340+0000 7ff7977fe640 10 monclient(hunting): handle_auth_done global_id 22123 payload 824 2024-05-15T13:20:25.343 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.340+0000 7ff7977fe640 1 -- 172.21.15.117:0/3890962144 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff79813c230 msgr2=0x7ff798135610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:25.343 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.340+0000 7ff7977fe640 1 --2- 172.21.15.117:0/3890962144 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff79813c230 0x7ff798135610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:25.343 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.340+0000 7ff7977fe640 1 -- 172.21.15.117:0/3890962144 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff798146100 msgr2=0x7ff798135b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:25.343 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.340+0000 7ff7977fe640 1 --2- 172.21.15.117:0/3890962144 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff798146100 0x7ff798135b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:25.343 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.340+0000 7ff7977fe640 10 monclient: _finish_hunting 0 2024-05-15T13:20:25.344 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.340+0000 7ff7977fe640 1 monclient: found mon.noname-b 2024-05-15T13:20:25.344 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.340+0000 7ff7977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:20:25.344 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.340+0000 7ff7977fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2024-05-15T13:20:25.344 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.340+0000 7ff7977fe640 1 -- 172.21.15.117:0/3890962144 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7981cde80 con 0x7ff798066cc0 2024-05-15T13:20:25.346 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.342+0000 7ff7977fe640 1 --2- 172.21.15.117:0/3890962144 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff798066cc0 0x7ff7981cc770 secure :-1 s=READY pgs=5437 cs=0 l=1 rev1=1 crypto rx=0x7ff78800e9e0 tx=0x7ff78800eea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:25.346 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.342+0000 7ff78ffff640 10 client.?.objecter ms_handle_connect 0x7ff798066cc0 2024-05-15T13:20:25.346 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.342+0000 7ff78ffff640 10 client.?.objecter resend_mon_ops 2024-05-15T13:20:25.346 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.346+0000 7ff78ffff640 1 -- 172.21.15.117:0/3890962144 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff788017070 con 0x7ff798066cc0 2024-05-15T13:20:25.346 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.346+0000 7ff78ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:25.346 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.346+0000 7ff78ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T13:20:25.346 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.346+0000 7ff78ffff640 10 monclient: dump: 2024-05-15T13:20:25.346 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:25.346 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:25.346 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:25.346 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:25.346 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:25.346 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:25.346 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:25.347 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:25.347 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:25.347 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:25.347 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.346+0000 7ff78ffff640 10 monclient: _finish_auth 0 2024-05-15T13:20:25.347 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.346+0000 7ff78ffff640 10 monclient: _check_auth_tickets 2024-05-15T13:20:25.347 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.346+0000 7ff78ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:20:25.347 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.346+0000 7ff78ffff640 1 -- 172.21.15.117:0/3890962144 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff78800c9d0 con 0x7ff798066cc0 2024-05-15T13:20:25.347 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.346+0000 7ff79e1ba640 5 monclient: authenticate success, global_id 22123 2024-05-15T13:20:25.347 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.346+0000 7ff79e1ba640 10 monclient: _renew_subs 2024-05-15T13:20:25.347 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.346+0000 7ff79e1ba640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T13:20:25.347 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.346+0000 7ff79e1ba640 1 -- 172.21.15.117:0/3890962144 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7981ce790 con 0x7ff798066cc0 2024-05-15T13:20:25.347 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.346+0000 7ff78ffff640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:20:25.347 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.346+0000 7ff78ffff640 1 -- 172.21.15.117:0/3890962144 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff78800cdc0 con 0x7ff798066cc0 2024-05-15T13:20:25.347 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.346+0000 7ff78ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:25.347 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.346+0000 7ff78ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T13:20:25.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.346+0000 7ff78ffff640 10 monclient: dump: 2024-05-15T13:20:25.348 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:25.348 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:25.348 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:25.348 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:25.348 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:25.348 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:25.348 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:25.348 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:25.348 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:25.348 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:25.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.346+0000 7ff79e1ba640 10 client.22123.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:20:25.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.346+0000 7ff7957fa640 4 set_mon_vals no callback set 2024-05-15T13:20:25.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.346+0000 7ff79e1ba640 10 monclient: _renew_subs 2024-05-15T13:20:25.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.346+0000 7ff79e1ba640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T13:20:25.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.346+0000 7ff79e1ba640 1 -- 172.21.15.117:0/3890962144 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7981cce90 con 0x7ff798066cc0 2024-05-15T13:20:25.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.346+0000 7ff79e1ba640 1 librados: init done 2024-05-15T13:20:25.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.347+0000 7ff79e1ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T13:20:25.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.347+0000 7ff79e1ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:20:25.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.347+0000 7ff79e1ba640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T13:20:25.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.347+0000 7ff79e1ba640 1 -- 172.21.15.117:0/3890962144 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff79804f0e0 con 0x7ff798066cc0 2024-05-15T13:20:25.591 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.591+0000 7ff78ffff640 1 -- 172.21.15.117:0/3890962144 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7ff78801a070 con 0x7ff798066cc0 2024-05-15T13:20:25.591 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.592+0000 7ff78ffff640 1 --2- 172.21.15.117:0/3890962144 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7ff77806ceb0 0x7ff77806f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:25.591 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.592+0000 7ff796ffd640 1 --2- 172.21.15.117:0/3890962144 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7ff77806ceb0 0x7ff77806f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:25.591 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.592+0000 7ff796ffd640 10 monclient: get_auth_request con 0x7ff77806ceb0 auth_method 0 2024-05-15T13:20:25.591 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.592+0000 7ff78ffff640 10 client.22123.objecter ms_handle_connect 0x7ff77806ceb0 2024-05-15T13:20:25.591 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.592+0000 7ff796ffd640 1 --2- 172.21.15.117:0/3890962144 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7ff77806ceb0 0x7ff77806f370 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7ff780036000 tx=0x7ff78003d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:25.594 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.595+0000 7ff78ffff640 1 -- 172.21.15.117:0/3890962144 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(1224..1224 src has 912..1224) v4 ==== 6365+0+0 (secure 0 0 0) 0x7ff788057220 con 0x7ff798066cc0 2024-05-15T13:20:25.594 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.595+0000 7ff78ffff640 10 client.22123.objecter ms_dispatch 0x7ff798000b90 osd_map(1224..1224 src has 912..1224) v4 2024-05-15T13:20:25.594 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.595+0000 7ff78ffff640 3 client.22123.objecter handle_osd_map got epochs [1224,1224] > 0 2024-05-15T13:20:25.594 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.595+0000 7ff78ffff640 3 client.22123.objecter handle_osd_map decoding full epoch 1224 2024-05-15T13:20:25.594 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.595+0000 7ff78ffff640 20 client.22123.objecter dump_active .. 0 homeless 2024-05-15T13:20:25.604 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.606+0000 7ff78ffff640 1 -- 172.21.15.117:0/3890962144 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7ff788059180 con 0x7ff798066cc0 2024-05-15T13:20:25.604 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.606+0000 7ff78ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:20:25.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.606+0000 7ff78ffff640 10 monclient: _finish_command 1 = system:0 2024-05-15T13:20:25.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.777+0000 7ff79e1ba640 1 -- 172.21.15.117:0/3890962144 --> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff7981ce310 con 0x7ff77806ceb0 2024-05-15T13:20:25.779 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.780+0000 7ff78ffff640 1 -- 172.21.15.117:0/3890962144 <== mgr.4107 v2:172.21.15.117:6800/1785766523 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+116276 (secure 0 0 0) 0x7ff780037040 con 0x7ff77806ceb0 2024-05-15T13:20:25.780 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T13:20:25.781 INFO:teuthology.orchestra.run.smithi117.stderr:dumped all 2024-05-15T13:20:25.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.783+0000 7ff79e1ba640 10 librados: watch_flush enter 2024-05-15T13:20:25.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.783+0000 7ff79e1ba640 10 librados: watch_flush exit 2024-05-15T13:20:25.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.783+0000 7ff79e1ba640 20 client.22123.objecter shutdown clearing up homeless session... 2024-05-15T13:20:25.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.783+0000 7ff79e1ba640 10 client.22123.objecter successfully canceled tick 2024-05-15T13:20:25.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.783+0000 7ff79e1ba640 1 -- 172.21.15.117:0/3890962144 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7ff77806ceb0 msgr2=0x7ff77806f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:25.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.783+0000 7ff79e1ba640 1 --2- 172.21.15.117:0/3890962144 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7ff77806ceb0 0x7ff77806f370 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7ff780036000 tx=0x7ff78003d040 comp rx=0 tx=0).stop 2024-05-15T13:20:25.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.783+0000 7ff79e1ba640 10 monclient: shutdown 2024-05-15T13:20:25.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.783+0000 7ff79e1ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:20:25.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.783+0000 7ff79e1ba640 1 -- 172.21.15.117:0/3890962144 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff798066cc0 msgr2=0x7ff7981cc770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:25.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.783+0000 7ff79e1ba640 1 --2- 172.21.15.117:0/3890962144 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff798066cc0 0x7ff7981cc770 secure :-1 s=READY pgs=5437 cs=0 l=1 rev1=1 crypto rx=0x7ff78800e9e0 tx=0x7ff78800eea0 comp rx=0 tx=0).stop 2024-05-15T13:20:25.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.783+0000 7ff79e1ba640 1 -- 172.21.15.117:0/3890962144 shutdown_connections 2024-05-15T13:20:25.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.783+0000 7ff79e1ba640 1 --2- 172.21.15.117:0/3890962144 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff798146100 0x7ff798135b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:25.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.783+0000 7ff79e1ba640 1 --2- 172.21.15.117:0/3890962144 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff79813c230 0x7ff798135610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:25.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.783+0000 7ff79e1ba640 1 --2- 172.21.15.117:0/3890962144 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7ff77806ceb0 0x7ff77806f370 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:25.784 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.783+0000 7ff79e1ba640 1 --2- 172.21.15.117:0/3890962144 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff798066cc0 0x7ff7981cc770 unknown :-1 s=CLOSED pgs=5437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:25.784 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.783+0000 7ff79e1ba640 1 -- 172.21.15.117:0/3890962144 >> 172.21.15.117:0/3890962144 conn(0x7ff7980b4a60 msgr2=0x7ff79813fac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:20:25.784 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.783+0000 7ff79e1ba640 1 -- 172.21.15.117:0/3890962144 shutdown_connections 2024-05-15T13:20:25.784 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.783+0000 7ff79e1ba640 1 -- 172.21.15.117:0/3890962144 wait complete. 2024-05-15T13:20:25.784 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.783+0000 7ff79e1ba640 1 librados: shutdown 2024-05-15T13:20:25.793 INFO:teuthology.orchestra.run.smithi117.stdout:{"pg_ready":true,"pg_map":{"version":2355,"stamp":"2024-05-15T13:20:24.612592+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":85,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":602048,"kb_used_data":41680,"kb_used_omap":12,"kb_used_meta":560243,"kb_avail":754372672,"statfs":{"total":773094113280,"available":772477616128,"internally_reserved":0,"allocated":42680320,"data_stored":21281579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":573689394},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":141,"apply_latency_ms":141,"commit_latency_ns":141000000,"apply_latency_ns":141000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.009369"},"pg_stats":[{"pgid":"6.16","version":"0'0","reported_seq":2354,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.597416+0000","last_change":"2024-05-15T12:44:59.567389+0000","last_active":"2024-05-15T13:20:19.597416+0000","last_peered":"2024-05-15T13:20:19.597416+0000","last_clean":"2024-05-15T13:20:19.597416+0000","last_became_active":"2024-05-15T12:44:59.550282+0000","last_became_peered":"2024-05-15T12:44:59.550282+0000","last_unstale":"2024-05-15T13:20:19.597416+0000","last_undegraded":"2024-05-15T13:20:19.597416+0000","last_fullsized":"2024-05-15T13:20:19.597416+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:44:22.143842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":2369,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.612783+0000","last_change":"2024-05-15T12:44:51.243387+0000","last_active":"2024-05-15T13:20:19.612783+0000","last_peered":"2024-05-15T13:20:19.612783+0000","last_clean":"2024-05-15T13:20:19.612783+0000","last_became_active":"2024-05-15T12:44:51.202412+0000","last_became_peered":"2024-05-15T12:44:51.202412+0000","last_unstale":"2024-05-15T13:20:19.612783+0000","last_undegraded":"2024-05-15T13:20:19.612783+0000","last_fullsized":"2024-05-15T13:20:19.612783+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:46:23.223713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":2355,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.590005+0000","last_change":"2024-05-15T12:45:13.532350+0000","last_active":"2024-05-15T13:20:19.590005+0000","last_peered":"2024-05-15T13:20:19.590005+0000","last_clean":"2024-05-15T13:20:19.590005+0000","last_became_active":"2024-05-15T12:45:11.468881+0000","last_became_peered":"2024-05-15T12:45:11.468881+0000","last_unstale":"2024-05-15T13:20:19.590005+0000","last_undegraded":"2024-05-15T13:20:19.590005+0000","last_fullsized":"2024-05-15T13:20:19.590005+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T15:53:17.739425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":2357,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.512008+0000","last_change":"2024-05-15T12:44:55.797152+0000","last_active":"2024-05-15T13:20:19.512008+0000","last_peered":"2024-05-15T13:20:19.512008+0000","last_clean":"2024-05-15T13:20:19.512008+0000","last_became_active":"2024-05-15T12:44:55.775514+0000","last_became_peered":"2024-05-15T12:44:55.775514+0000","last_unstale":"2024-05-15T13:20:19.512008+0000","last_undegraded":"2024-05-15T13:20:19.512008+0000","last_fullsized":"2024-05-15T13:20:19.512008+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:09:17.199545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":2355,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.614535+0000","last_change":"2024-05-15T12:44:57.700224+0000","last_active":"2024-05-15T13:20:19.614535+0000","last_peered":"2024-05-15T13:20:19.614535+0000","last_clean":"2024-05-15T13:20:19.614535+0000","last_became_active":"2024-05-15T12:44:57.673614+0000","last_became_peered":"2024-05-15T12:44:57.673614+0000","last_unstale":"2024-05-15T13:20:19.614535+0000","last_undegraded":"2024-05-15T13:20:19.614535+0000","last_fullsized":"2024-05-15T13:20:19.614535+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-17T00:09:31.317495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":2355,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.583562+0000","last_change":"2024-05-15T12:44:59.486234+0000","last_active":"2024-05-15T13:20:19.583562+0000","last_peered":"2024-05-15T13:20:19.583562+0000","last_clean":"2024-05-15T13:20:19.583562+0000","last_became_active":"2024-05-15T12:44:59.480451+0000","last_became_peered":"2024-05-15T12:44:59.480451+0000","last_unstale":"2024-05-15T13:20:19.583562+0000","last_undegraded":"2024-05-15T13:20:19.583562+0000","last_fullsized":"2024-05-15T13:20:19.583562+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:28:22.234323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":2361,"reported_epoch":1221,"state":"active+clean","last_fresh":"2024-05-15T13:20:20.722753+0000","last_change":"2024-05-15T12:44:54.924153+0000","last_active":"2024-05-15T13:20:20.722753+0000","last_peered":"2024-05-15T13:20:20.722753+0000","last_clean":"2024-05-15T13:20:20.722753+0000","last_became_active":"2024-05-15T12:44:54.914910+0000","last_became_peered":"2024-05-15T12:44:54.914910+0000","last_unstale":"2024-05-15T13:20:20.722753+0000","last_undegraded":"2024-05-15T13:20:20.722753+0000","last_fullsized":"2024-05-15T13:20:20.722753+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-17T00:36:49.391069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":2356,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.564063+0000","last_change":"2024-05-15T12:44:59.650327+0000","last_active":"2024-05-15T13:20:19.564063+0000","last_peered":"2024-05-15T13:20:19.564063+0000","last_clean":"2024-05-15T13:20:19.564063+0000","last_became_active":"2024-05-15T12:44:59.595062+0000","last_became_peered":"2024-05-15T12:44:59.595062+0000","last_unstale":"2024-05-15T13:20:19.564063+0000","last_undegraded":"2024-05-15T13:20:19.564063+0000","last_fullsized":"2024-05-15T13:20:19.564063+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T12:52:31.383828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":2371,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.607147+0000","last_change":"2024-05-15T12:44:51.802252+0000","last_active":"2024-05-15T13:20:19.607147+0000","last_peered":"2024-05-15T13:20:19.607147+0000","last_clean":"2024-05-15T13:20:19.607147+0000","last_became_active":"2024-05-15T12:44:51.785998+0000","last_became_peered":"2024-05-15T12:44:51.785998+0000","last_unstale":"2024-05-15T13:20:19.607147+0000","last_undegraded":"2024-05-15T13:20:19.607147+0000","last_fullsized":"2024-05-15T13:20:19.607147+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:00:30.969147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":2372,"reported_epoch":1221,"state":"active+clean","last_fresh":"2024-05-15T13:20:20.670827+0000","last_change":"2024-05-15T12:44:53.614201+0000","last_active":"2024-05-15T13:20:20.670827+0000","last_peered":"2024-05-15T13:20:20.670827+0000","last_clean":"2024-05-15T13:20:20.670827+0000","last_became_active":"2024-05-15T12:44:53.608407+0000","last_became_peered":"2024-05-15T12:44:53.608407+0000","last_unstale":"2024-05-15T13:20:20.670827+0000","last_undegraded":"2024-05-15T13:20:20.670827+0000","last_fullsized":"2024-05-15T13:20:20.670827+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:45:19.480038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":2362,"reported_epoch":1222,"state":"active+clean","last_fresh":"2024-05-15T13:20:22.298535+0000","last_change":"2024-05-15T12:44:52.221672+0000","last_active":"2024-05-15T13:20:22.298535+0000","last_peered":"2024-05-15T13:20:22.298535+0000","last_clean":"2024-05-15T13:20:22.298535+0000","last_became_active":"2024-05-15T12:44:52.212480+0000","last_became_peered":"2024-05-15T12:44:52.212480+0000","last_unstale":"2024-05-15T13:20:22.298535+0000","last_undegraded":"2024-05-15T13:20:22.298535+0000","last_fullsized":"2024-05-15T13:20:22.298535+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:41:34.164739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":2371,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.609138+0000","last_change":"2024-05-15T12:44:56.446419+0000","last_active":"2024-05-15T13:20:19.609138+0000","last_peered":"2024-05-15T13:20:19.609138+0000","last_clean":"2024-05-15T13:20:19.609138+0000","last_became_active":"2024-05-15T12:44:56.197837+0000","last_became_peered":"2024-05-15T12:44:56.197837+0000","last_unstale":"2024-05-15T13:20:19.609138+0000","last_undegraded":"2024-05-15T13:20:19.609138+0000","last_fullsized":"2024-05-15T13:20:19.609138+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:33:22.475565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":2360,"reported_epoch":1221,"state":"active+clean","last_fresh":"2024-05-15T13:20:20.814121+0000","last_change":"2024-05-15T12:44:53.399230+0000","last_active":"2024-05-15T13:20:20.814121+0000","last_peered":"2024-05-15T13:20:20.814121+0000","last_clean":"2024-05-15T13:20:20.814121+0000","last_became_active":"2024-05-15T12:44:53.377773+0000","last_became_peered":"2024-05-15T12:44:53.377773+0000","last_unstale":"2024-05-15T13:20:20.814121+0000","last_undegraded":"2024-05-15T13:20:20.814121+0000","last_fullsized":"2024-05-15T13:20:20.814121+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:12:39.424987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'76","reported_seq":2393,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.589082+0000","last_change":"2024-05-15T12:43:04.763767+0000","last_active":"2024-05-15T13:20:19.589082+0000","last_peered":"2024-05-15T13:20:19.589082+0000","last_clean":"2024-05-15T13:20:19.589082+0000","last_became_active":"2024-05-15T12:43:04.148735+0000","last_became_peered":"2024-05-15T12:43:04.148735+0000","last_unstale":"2024-05-15T13:20:19.589082+0000","last_undegraded":"2024-05-15T13:20:19.589082+0000","last_fullsized":"2024-05-15T13:20:19.589082+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:42:53.071197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:42:53.071197+0000","last_clean_scrub_stamp":"2024-05-15T12:42:53.071197+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:45:23.691461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2370,"reported_epoch":1221,"state":"active+clean","last_fresh":"2024-05-15T13:20:20.778978+0000","last_change":"2024-05-15T12:43:09.453018+0000","last_active":"2024-05-15T13:20:20.778978+0000","last_peered":"2024-05-15T13:20:20.778978+0000","last_clean":"2024-05-15T13:20:20.778978+0000","last_became_active":"2024-05-15T12:43:09.445878+0000","last_became_peered":"2024-05-15T12:43:09.445878+0000","last_unstale":"2024-05-15T13:20:20.778978+0000","last_undegraded":"2024-05-15T13:20:20.778978+0000","last_fullsized":"2024-05-15T13:20:20.778978+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_clean_scrub_stamp":"2024-05-15T12:43:07.133352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:55:05.766298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":2364,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.629451+0000","last_change":"2024-05-15T12:44:50.705660+0000","last_active":"2024-05-15T13:20:19.629451+0000","last_peered":"2024-05-15T13:20:19.629451+0000","last_clean":"2024-05-15T13:20:19.629451+0000","last_became_active":"2024-05-15T12:44:50.696664+0000","last_became_peered":"2024-05-15T12:44:50.696664+0000","last_unstale":"2024-05-15T13:20:19.629451+0000","last_undegraded":"2024-05-15T13:20:19.629451+0000","last_fullsized":"2024-05-15T13:20:19.629451+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:15:31.819701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2371,"reported_epoch":1221,"state":"active+clean","last_fresh":"2024-05-15T13:20:20.720837+0000","last_change":"2024-05-15T12:43:08.704425+0000","last_active":"2024-05-15T13:20:20.720837+0000","last_peered":"2024-05-15T13:20:20.720837+0000","last_clean":"2024-05-15T13:20:20.720837+0000","last_became_active":"2024-05-15T12:43:08.677751+0000","last_became_peered":"2024-05-15T12:43:08.677751+0000","last_unstale":"2024-05-15T13:20:20.720837+0000","last_undegraded":"2024-05-15T13:20:20.720837+0000","last_fullsized":"2024-05-15T13:20:20.720837+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_clean_scrub_stamp":"2024-05-15T12:43:07.133352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-17T00:22:51.843061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":2355,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.638687+0000","last_change":"2024-05-15T12:44:57.579283+0000","last_active":"2024-05-15T13:20:19.638687+0000","last_peered":"2024-05-15T13:20:19.638687+0000","last_clean":"2024-05-15T13:20:19.638687+0000","last_became_active":"2024-05-15T12:44:57.552111+0000","last_became_peered":"2024-05-15T12:44:57.552111+0000","last_unstale":"2024-05-15T13:20:19.638687+0000","last_undegraded":"2024-05-15T13:20:19.638687+0000","last_fullsized":"2024-05-15T13:20:19.638687+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:23:14.388995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2372,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.618288+0000","last_change":"2024-05-15T12:43:08.634999+0000","last_active":"2024-05-15T13:20:19.618288+0000","last_peered":"2024-05-15T13:20:19.618288+0000","last_clean":"2024-05-15T13:20:19.618288+0000","last_became_active":"2024-05-15T12:43:08.600745+0000","last_became_peered":"2024-05-15T12:43:08.600745+0000","last_unstale":"2024-05-15T13:20:19.618288+0000","last_undegraded":"2024-05-15T13:20:19.618288+0000","last_fullsized":"2024-05-15T13:20:19.618288+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_clean_scrub_stamp":"2024-05-15T12:43:07.133352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:17:56.643544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":2373,"reported_epoch":1221,"state":"active+clean","last_fresh":"2024-05-15T13:20:20.656371+0000","last_change":"2024-05-15T12:44:51.361084+0000","last_active":"2024-05-15T13:20:20.656371+0000","last_peered":"2024-05-15T13:20:20.656371+0000","last_clean":"2024-05-15T13:20:20.656371+0000","last_became_active":"2024-05-15T12:44:51.354879+0000","last_became_peered":"2024-05-15T12:44:51.354879+0000","last_unstale":"2024-05-15T13:20:20.656371+0000","last_undegraded":"2024-05-15T13:20:20.656371+0000","last_fullsized":"2024-05-15T13:20:20.656371+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:53:33.707185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2370,"reported_epoch":1221,"state":"active+clean","last_fresh":"2024-05-15T13:20:20.765250+0000","last_change":"2024-05-15T12:43:09.375465+0000","last_active":"2024-05-15T13:20:20.765250+0000","last_peered":"2024-05-15T13:20:20.765250+0000","last_clean":"2024-05-15T13:20:20.765250+0000","last_became_active":"2024-05-15T12:43:09.363876+0000","last_became_peered":"2024-05-15T12:43:09.363876+0000","last_unstale":"2024-05-15T13:20:20.765250+0000","last_undegraded":"2024-05-15T13:20:20.765250+0000","last_fullsized":"2024-05-15T13:20:20.765250+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_clean_scrub_stamp":"2024-05-15T12:43:07.133352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:27:46.599385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":2364,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.594769+0000","last_change":"2024-05-15T12:44:54.188435+0000","last_active":"2024-05-15T13:20:19.594769+0000","last_peered":"2024-05-15T13:20:19.594769+0000","last_clean":"2024-05-15T13:20:19.594769+0000","last_became_active":"2024-05-15T12:44:54.179460+0000","last_became_peered":"2024-05-15T12:44:54.179460+0000","last_unstale":"2024-05-15T13:20:19.594769+0000","last_undegraded":"2024-05-15T13:20:19.594769+0000","last_fullsized":"2024-05-15T13:20:19.594769+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T12:52:27.875820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2382,"reported_epoch":1221,"state":"active+clean","last_fresh":"2024-05-15T13:20:20.653740+0000","last_change":"2024-05-15T12:43:09.384765+0000","last_active":"2024-05-15T13:20:20.653740+0000","last_peered":"2024-05-15T13:20:20.653740+0000","last_clean":"2024-05-15T13:20:20.653740+0000","last_became_active":"2024-05-15T12:43:09.377725+0000","last_became_peered":"2024-05-15T12:43:09.377725+0000","last_unstale":"2024-05-15T13:20:20.653740+0000","last_undegraded":"2024-05-15T13:20:20.653740+0000","last_fullsized":"2024-05-15T13:20:20.653740+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_clean_scrub_stamp":"2024-05-15T12:43:07.133352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:57:59.671602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"47'7","reported_seq":2377,"reported_epoch":1222,"state":"active+clean","last_fresh":"2024-05-15T13:20:22.279258+0000","last_change":"2024-05-15T12:44:58.436179+0000","last_active":"2024-05-15T13:20:22.279258+0000","last_peered":"2024-05-15T13:20:22.279258+0000","last_clean":"2024-05-15T13:20:22.279258+0000","last_became_active":"2024-05-15T12:44:58.318306+0000","last_became_peered":"2024-05-15T12:44:58.318306+0000","last_unstale":"2024-05-15T13:20:22.279258+0000","last_undegraded":"2024-05-15T13:20:22.279258+0000","last_fullsized":"2024-05-15T13:20:22.279258+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:59:46.310559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2370,"reported_epoch":1221,"state":"active+clean","last_fresh":"2024-05-15T13:20:20.794871+0000","last_change":"2024-05-15T12:43:08.778833+0000","last_active":"2024-05-15T13:20:20.794871+0000","last_peered":"2024-05-15T13:20:20.794871+0000","last_clean":"2024-05-15T13:20:20.794871+0000","last_became_active":"2024-05-15T12:43:08.773614+0000","last_became_peered":"2024-05-15T12:43:08.773614+0000","last_unstale":"2024-05-15T13:20:20.794871+0000","last_undegraded":"2024-05-15T13:20:20.794871+0000","last_fullsized":"2024-05-15T13:20:20.794871+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_clean_scrub_stamp":"2024-05-15T12:43:07.133352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:22:59.785687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":2361,"reported_epoch":1221,"state":"active+clean","last_fresh":"2024-05-15T13:20:20.771718+0000","last_change":"2024-05-15T12:45:08.669407+0000","last_active":"2024-05-15T13:20:20.771718+0000","last_peered":"2024-05-15T13:20:20.771718+0000","last_clean":"2024-05-15T13:20:20.771718+0000","last_became_active":"2024-05-15T12:45:08.661140+0000","last_became_peered":"2024-05-15T12:45:08.661140+0000","last_unstale":"2024-05-15T13:20:20.771718+0000","last_undegraded":"2024-05-15T13:20:20.771718+0000","last_fullsized":"2024-05-15T13:20:20.771718+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:55:43.700115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2366,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.599275+0000","last_change":"2024-05-15T12:43:09.478046+0000","last_active":"2024-05-15T13:20:19.599275+0000","last_peered":"2024-05-15T13:20:19.599275+0000","last_clean":"2024-05-15T13:20:19.599275+0000","last_became_active":"2024-05-15T12:43:09.448237+0000","last_became_peered":"2024-05-15T12:43:09.448237+0000","last_unstale":"2024-05-15T13:20:19.599275+0000","last_undegraded":"2024-05-15T13:20:19.599275+0000","last_fullsized":"2024-05-15T13:20:19.599275+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_clean_scrub_stamp":"2024-05-15T12:43:07.133352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:11:17.055288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":2355,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.612460+0000","last_change":"2024-05-15T12:45:18.657708+0000","last_active":"2024-05-15T13:20:19.612460+0000","last_peered":"2024-05-15T13:20:19.612460+0000","last_clean":"2024-05-15T13:20:19.612460+0000","last_became_active":"2024-05-15T12:45:18.642568+0000","last_became_peered":"2024-05-15T12:45:18.642568+0000","last_unstale":"2024-05-15T13:20:19.612460+0000","last_undegraded":"2024-05-15T13:20:19.612460+0000","last_fullsized":"2024-05-15T13:20:19.612460+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:21:30.095711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2369,"reported_epoch":1221,"state":"active+clean","last_fresh":"2024-05-15T13:20:20.815988+0000","last_change":"2024-05-15T12:43:09.429828+0000","last_active":"2024-05-15T13:20:20.815988+0000","last_peered":"2024-05-15T13:20:20.815988+0000","last_clean":"2024-05-15T13:20:20.815988+0000","last_became_active":"2024-05-15T12:43:09.424460+0000","last_became_peered":"2024-05-15T12:43:09.424460+0000","last_unstale":"2024-05-15T13:20:20.815988+0000","last_undegraded":"2024-05-15T13:20:20.815988+0000","last_fullsized":"2024-05-15T13:20:20.815988+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_clean_scrub_stamp":"2024-05-15T12:43:07.133352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:12:21.424590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":2362,"reported_epoch":1222,"state":"active+clean","last_fresh":"2024-05-15T13:20:22.277210+0000","last_change":"2024-05-15T12:44:52.129096+0000","last_active":"2024-05-15T13:20:22.277210+0000","last_peered":"2024-05-15T13:20:22.277210+0000","last_clean":"2024-05-15T13:20:22.277210+0000","last_became_active":"2024-05-15T12:44:52.088745+0000","last_became_peered":"2024-05-15T12:44:52.088745+0000","last_unstale":"2024-05-15T13:20:22.277210+0000","last_undegraded":"2024-05-15T13:20:22.277210+0000","last_fullsized":"2024-05-15T13:20:22.277210+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:16:34.966002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":2355,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.552600+0000","last_change":"2024-05-15T12:44:56.480904+0000","last_active":"2024-05-15T13:20:19.552600+0000","last_peered":"2024-05-15T13:20:19.552600+0000","last_clean":"2024-05-15T13:20:19.552600+0000","last_became_active":"2024-05-15T12:44:56.470246+0000","last_became_peered":"2024-05-15T12:44:56.470246+0000","last_unstale":"2024-05-15T13:20:19.552600+0000","last_undegraded":"2024-05-15T13:20:19.552600+0000","last_fullsized":"2024-05-15T13:20:19.552600+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:53:59.488101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":2361,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.610996+0000","last_change":"2024-05-15T12:44:51.677390+0000","last_active":"2024-05-15T13:20:19.610996+0000","last_peered":"2024-05-15T13:20:19.610996+0000","last_clean":"2024-05-15T13:20:19.610996+0000","last_became_active":"2024-05-15T12:44:51.660527+0000","last_became_peered":"2024-05-15T12:44:51.660527+0000","last_unstale":"2024-05-15T13:20:19.610996+0000","last_undegraded":"2024-05-15T13:20:19.610996+0000","last_fullsized":"2024-05-15T13:20:19.610996+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:25:49.920807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":2354,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.616476+0000","last_change":"2024-05-15T12:45:12.010393+0000","last_active":"2024-05-15T13:20:19.616476+0000","last_peered":"2024-05-15T13:20:19.616476+0000","last_clean":"2024-05-15T13:20:19.616476+0000","last_became_active":"2024-05-15T12:45:11.998256+0000","last_became_peered":"2024-05-15T12:45:11.998256+0000","last_unstale":"2024-05-15T13:20:19.616476+0000","last_undegraded":"2024-05-15T13:20:19.616476+0000","last_fullsized":"2024-05-15T13:20:19.616476+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:13:05.556724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":2360,"reported_epoch":1222,"state":"active+clean","last_fresh":"2024-05-15T13:20:22.317768+0000","last_change":"2024-05-15T12:44:55.564951+0000","last_active":"2024-05-15T13:20:22.317768+0000","last_peered":"2024-05-15T13:20:22.317768+0000","last_clean":"2024-05-15T13:20:22.317768+0000","last_became_active":"2024-05-15T12:44:55.557214+0000","last_became_peered":"2024-05-15T12:44:55.557214+0000","last_unstale":"2024-05-15T13:20:22.317768+0000","last_undegraded":"2024-05-15T13:20:22.317768+0000","last_fullsized":"2024-05-15T13:20:22.317768+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:53:15.834038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":2359,"reported_epoch":1221,"state":"active+clean","last_fresh":"2024-05-15T13:20:20.785961+0000","last_change":"2024-05-15T12:44:56.078343+0000","last_active":"2024-05-15T13:20:20.785961+0000","last_peered":"2024-05-15T13:20:20.785961+0000","last_clean":"2024-05-15T13:20:20.785961+0000","last_became_active":"2024-05-15T12:44:56.072618+0000","last_became_peered":"2024-05-15T12:44:56.072618+0000","last_unstale":"2024-05-15T13:20:20.785961+0000","last_undegraded":"2024-05-15T13:20:20.785961+0000","last_fullsized":"2024-05-15T13:20:20.785961+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:22:00.980595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":2364,"reported_epoch":1222,"state":"active+clean","last_fresh":"2024-05-15T13:20:22.332975+0000","last_change":"2024-05-15T12:44:55.746501+0000","last_active":"2024-05-15T13:20:22.332975+0000","last_peered":"2024-05-15T13:20:22.332975+0000","last_clean":"2024-05-15T13:20:22.332975+0000","last_became_active":"2024-05-15T12:44:55.739811+0000","last_became_peered":"2024-05-15T12:44:55.739811+0000","last_unstale":"2024-05-15T13:20:22.332975+0000","last_undegraded":"2024-05-15T13:20:22.332975+0000","last_fullsized":"2024-05-15T13:20:22.332975+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T12:49:40.051745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":2361,"reported_epoch":1221,"state":"active+clean","last_fresh":"2024-05-15T13:20:20.800642+0000","last_change":"2024-05-15T12:44:56.495925+0000","last_active":"2024-05-15T13:20:20.800642+0000","last_peered":"2024-05-15T13:20:20.800642+0000","last_clean":"2024-05-15T13:20:20.800642+0000","last_became_active":"2024-05-15T12:44:56.429996+0000","last_became_peered":"2024-05-15T12:44:56.429996+0000","last_unstale":"2024-05-15T13:20:20.800642+0000","last_undegraded":"2024-05-15T13:20:20.800642+0000","last_fullsized":"2024-05-15T13:20:20.800642+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:18:31.842665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":2353,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.604742+0000","last_change":"2024-05-15T12:45:18.556824+0000","last_active":"2024-05-15T13:20:19.604742+0000","last_peered":"2024-05-15T13:20:19.604742+0000","last_clean":"2024-05-15T13:20:19.604742+0000","last_became_active":"2024-05-15T12:45:18.546529+0000","last_became_peered":"2024-05-15T12:45:18.546529+0000","last_unstale":"2024-05-15T13:20:19.604742+0000","last_undegraded":"2024-05-15T13:20:19.604742+0000","last_fullsized":"2024-05-15T13:20:19.604742+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:07:19.212044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":2354,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.555166+0000","last_change":"2024-05-15T12:44:56.642476+0000","last_active":"2024-05-15T13:20:19.555166+0000","last_peered":"2024-05-15T13:20:19.555166+0000","last_clean":"2024-05-15T13:20:19.555166+0000","last_became_active":"2024-05-15T12:44:56.634257+0000","last_became_peered":"2024-05-15T12:44:56.634257+0000","last_unstale":"2024-05-15T13:20:19.555166+0000","last_undegraded":"2024-05-15T13:20:19.555166+0000","last_fullsized":"2024-05-15T13:20:19.555166+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T15:44:34.817095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":2361,"reported_epoch":1222,"state":"active+clean","last_fresh":"2024-05-15T13:20:22.287049+0000","last_change":"2024-05-15T12:44:52.504602+0000","last_active":"2024-05-15T13:20:22.287049+0000","last_peered":"2024-05-15T13:20:22.287049+0000","last_clean":"2024-05-15T13:20:22.287049+0000","last_became_active":"2024-05-15T12:44:52.483168+0000","last_became_peered":"2024-05-15T12:44:52.483168+0000","last_unstale":"2024-05-15T13:20:22.287049+0000","last_undegraded":"2024-05-15T13:20:22.287049+0000","last_fullsized":"2024-05-15T13:20:22.287049+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:17:24.835749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":2359,"reported_epoch":1221,"state":"active+clean","last_fresh":"2024-05-15T13:20:20.768531+0000","last_change":"2024-05-15T12:45:12.195717+0000","last_active":"2024-05-15T13:20:20.768531+0000","last_peered":"2024-05-15T13:20:20.768531+0000","last_clean":"2024-05-15T13:20:20.768531+0000","last_became_active":"2024-05-15T12:45:12.190202+0000","last_became_peered":"2024-05-15T12:45:12.190202+0000","last_unstale":"2024-05-15T13:20:20.768531+0000","last_undegraded":"2024-05-15T13:20:20.768531+0000","last_fullsized":"2024-05-15T13:20:20.768531+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:05:34.894666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":20,"seq":85899346373,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72784,"kb_used_data":5504,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299056,"statfs":{"total":96636764160,"available":96562233344,"internally_reserved":0,"allocated":5636096,"data_stored":2959141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[]},{"osd":0,"up_from":20,"seq":85899346372,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68220,"kb_used_data":5100,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303620,"statfs":{"total":96636764160,"available":96566906880,"internally_reserved":0,"allocated":5222400,"data_stored":2553456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[]},{"osd":2,"up_from":20,"seq":85899346372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94532,"kb_used_data":5112,"kb_used_omap":1,"kb_used_meta":89406,"kb_avail":94277308,"statfs":{"total":96636764160,"available":96539963392,"internally_reserved":0,"allocated":5234688,"data_stored":2561301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":91552198},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":6,"up_from":19,"seq":81604379076,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72404,"kb_used_data":5128,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299436,"statfs":{"total":96636764160,"available":96562622464,"internally_reserved":0,"allocated":5251072,"data_stored":2568766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[]},{"osd":4,"up_from":19,"seq":81604379076,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72376,"kb_used_data":5100,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299464,"statfs":{"total":96636764160,"available":96562651136,"internally_reserved":0,"allocated":5222400,"data_stored":2553456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509890,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72396,"kb_used_data":5116,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299444,"statfs":{"total":96636764160,"available":96562630656,"internally_reserved":0,"allocated":5238784,"data_stored":2561306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542595,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72796,"kb_used_data":5520,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299044,"statfs":{"total":96636764160,"available":96562221056,"internally_reserved":0,"allocated":5652480,"data_stored":2970697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[]},{"osd":7,"up_from":13,"seq":55834575300,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76540,"kb_used_data":5100,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94295300,"statfs":{"total":96636764160,"available":96558387200,"internally_reserved":0,"allocated":5222400,"data_stored":2553456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T13:20:25.795 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-05-15T13:20:25.795 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-15T13:20:25.893 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.895+0000 7fe3cff52640 10 monclient: get_monmap_and_config 2024-05-15T13:20:25.893 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.895+0000 7fe3cff52640 10 monclient: build_initial_monmap 2024-05-15T13:20:25.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.895+0000 7fe3cff52640 10 monclient: monmap: 2024-05-15T13:20:25.894 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T13:20:25.894 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:25.894 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T13:20:25.896141+0000 2024-05-15T13:20:25.894 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T13:20:25.896141+0000 2024-05-15T13:20:25.894 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T13:20:25.894 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:25.894 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T13:20:25.894 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T13:20:25.894 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T13:20:25.894 INFO:teuthology.orchestra.run.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T13:20:25.894 INFO:teuthology.orchestra.run.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T13:20:25.894 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:25.896 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.895+0000 7fe3cff52640 1 Processor -- start 2024-05-15T13:20:25.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.896+0000 7fe3cff52640 1 -- start start 2024-05-15T13:20:25.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.896+0000 7fe3cff52640 10 monclient: init 2024-05-15T13:20:25.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.896+0000 7fe3cff52640 10 monclient: _reopen_session rank -1 2024-05-15T13:20:25.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.896+0000 7fe3cff52640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-05-15T13:20:25.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.896+0000 7fe3cff52640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fe3c806bd70 0x7fe3c8142940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:25.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.896+0000 7fe3cff52640 10 monclient(hunting): picked mon.noname-a con 0x7fe3c806bd70 addr v2:172.21.15.117:3300/0 2024-05-15T13:20:25.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.896+0000 7fe3cff52640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe3c8070b80 0x7fe3c8072570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:25.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.896+0000 7fe3cff52640 10 monclient(hunting): picked mon.noname-b con 0x7fe3c8070b80 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T13:20:25.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.896+0000 7fe3cff52640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe3c8072ab0 addr v1:172.21.15.117:6789/0 2024-05-15T13:20:25.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.896+0000 7fe3cff52640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3c8071ec0 con 0x7fe3c8072ab0 2024-05-15T13:20:25.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.896+0000 7fe3cff52640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:25.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.896+0000 7fe3cff52640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3c806c150 con 0x7fe3c806bd70 2024-05-15T13:20:25.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.896+0000 7fe3cff52640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:25.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.896+0000 7fe3cff52640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c806fab0 con 0x7fe3c8070b80 2024-05-15T13:20:25.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.896+0000 7fe3cff52640 10 monclient(hunting): _renew_subs 2024-05-15T13:20:25.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.896+0000 7fe3cff52640 10 monclient(hunting): authenticate will time out at 3589.138457s 2024-05-15T13:20:25.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.897+0000 7fe3cd4ed640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe3c8070b80 0x7fe3c8072570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:25.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.897+0000 7fe3ce4ef640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fe3c8072ab0 0x7fe3c80ad820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:54076/0 (socket says 172.21.15.117:54076) 2024-05-15T13:20:25.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.897+0000 7fe3ce4ef640 1 -- 172.21.15.117:0/1187508908 learned_addr learned my addr 172.21.15.117:0/1187508908 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-05-15T13:20:25.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.897+0000 7fe3cdcee640 1 --2- 172.21.15.117:0/1187508908 >> v2:172.21.15.117:3300/0 conn(0x7fe3c806bd70 0x7fe3c8142940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:25.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.897+0000 7fe3cd4ed640 10 monclient(hunting): get_auth_request con 0x7fe3c8070b80 auth_method 0 2024-05-15T13:20:25.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.897+0000 7fe3cd4ed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:25.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.897+0000 7fe3cd4ed640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:25.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.897+0000 7fe3cd4ed640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:25.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.898+0000 7fe3cdcee640 10 monclient(hunting): get_auth_request con 0x7fe3c806bd70 auth_method 0 2024-05-15T13:20:25.899 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.898+0000 7fe3cdcee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:25.899 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.898+0000 7fe3cdcee640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:25.899 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.898+0000 7fe3cdcee640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:25.899 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.898+0000 7fe3cd4ed640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:25.899 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.898+0000 7fe3cd4ed640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:25.899 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.898+0000 7fe3cd4ed640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:25.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.898+0000 7fe3cdcee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:25.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.898+0000 7fe3cdcee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:25.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.898+0000 7fe3cdcee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:25.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.900+0000 7fe3cd4ed640 10 monclient(hunting): handle_auth_done global_id 18707 payload 274 2024-05-15T13:20:25.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.900+0000 7fe3cd4ed640 1 -- 172.21.15.117:0/1187508908 >> v2:172.21.15.117:3300/0 conn(0x7fe3c806bd70 msgr2=0x7fe3c8142940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T13:20:25.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.900+0000 7fe3cd4ed640 1 --2- 172.21.15.117:0/1187508908 >> v2:172.21.15.117:3300/0 conn(0x7fe3c806bd70 0x7fe3c8142940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:25.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.900+0000 7fe3cd4ed640 1 -- 172.21.15.117:0/1187508908 >> v1:172.21.15.117:6789/0 conn(0x7fe3c8072ab0 legacy=0x7fe3c80ad820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:25.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.900+0000 7fe3cd4ed640 10 monclient: _finish_hunting 0 2024-05-15T13:20:25.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.900+0000 7fe3cd4ed640 1 monclient: found mon.noname-b 2024-05-15T13:20:25.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.900+0000 7fe3cd4ed640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:20:25.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.900+0000 7fe3cd4ed640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.117:3301/0 2024-05-15T13:20:25.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.900+0000 7fe3cd4ed640 1 -- 172.21.15.117:0/1187508908 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3c813cf00 con 0x7fe3c8070b80 2024-05-15T13:20:25.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.902+0000 7fe3cd4ed640 1 --2- 172.21.15.117:0/1187508908 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe3c8070b80 0x7fe3c8072570 secure :-1 s=READY pgs=3379 cs=0 l=1 rev1=1 crypto rx=0x7fe3b80097a0 tx=0x7fe3b802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4519d7243290e9e2 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:25.906 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.906+0000 7fe3cccec640 1 -- 172.21.15.117:0/1187508908 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b8035020 con 0x7fe3c8070b80 2024-05-15T13:20:25.906 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.906+0000 7fe3cccec640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:25.906 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.906+0000 7fe3cccec640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T13:20:25.906 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.906+0000 7fe3cccec640 10 monclient: dump: 2024-05-15T13:20:25.906 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:25.906 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:25.906 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:25.906 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:25.906 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:25.907 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:25.907 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:25.907 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:25.907 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:25.907 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:25.907 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.906+0000 7fe3cccec640 10 monclient: _finish_auth 0 2024-05-15T13:20:25.907 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.906+0000 7fe3cccec640 10 monclient: _check_auth_tickets 2024-05-15T13:20:25.907 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.906+0000 7fe3cccec640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:20:25.907 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.906+0000 7fe3cccec640 1 -- 172.21.15.117:0/1187508908 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3b8031830 con 0x7fe3c8070b80 2024-05-15T13:20:25.907 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.906+0000 7fe3cff52640 5 monclient: authenticate success, global_id 18707 2024-05-15T13:20:25.907 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.906+0000 7fe3cff52640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T13:20:25.907 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.906+0000 7fe3cccec640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:20:25.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.906+0000 7fe3cff52640 10 monclient: get_monmap_and_config success 2024-05-15T13:20:25.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.906+0000 7fe3cff52640 4 set_mon_vals no callback set 2024-05-15T13:20:25.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.906+0000 7fe3cccec640 1 -- 172.21.15.117:0/1187508908 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b8031c00 con 0x7fe3c8070b80 2024-05-15T13:20:25.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.906+0000 7fe3cccec640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:25.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.906+0000 7fe3cccec640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T13:20:25.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.906+0000 7fe3cccec640 10 monclient: dump: 2024-05-15T13:20:25.908 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:25.908 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:25.908 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:25.908 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:25.908 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:25.908 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:25.908 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:25.908 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:25.909 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:25.909 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:25.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.906+0000 7fe3cff52640 10 monclient: shutdown 2024-05-15T13:20:25.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.906+0000 7fe3cff52640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:20:25.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.906+0000 7fe3cff52640 1 -- 172.21.15.117:0/1187508908 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe3c8070b80 msgr2=0x7fe3c8072570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:25.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.907+0000 7fe3cff52640 1 --2- 172.21.15.117:0/1187508908 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe3c8070b80 0x7fe3c8072570 secure :-1 s=READY pgs=3379 cs=0 l=1 rev1=1 crypto rx=0x7fe3b80097a0 tx=0x7fe3b802b8d0 comp rx=0 tx=0).stop 2024-05-15T13:20:25.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.907+0000 7fe3cff52640 1 -- 172.21.15.117:0/1187508908 shutdown_connections 2024-05-15T13:20:25.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.907+0000 7fe3cff52640 1 --2- 172.21.15.117:0/1187508908 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe3c8070b80 0x7fe3c8072570 unknown :-1 s=CLOSED pgs=3379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:25.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.907+0000 7fe3cff52640 1 --2- 172.21.15.117:0/1187508908 >> v2:172.21.15.117:3300/0 conn(0x7fe3c806bd70 0x7fe3c8142940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:25.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.907+0000 7fe3cff52640 1 -- 172.21.15.117:0/1187508908 >> 172.21.15.117:0/1187508908 conn(0x7fe3c806ada0 msgr2=0x7fe3c806b1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:20:25.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.907+0000 7fe3cff52640 1 -- 172.21.15.117:0/1187508908 shutdown_connections 2024-05-15T13:20:25.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.907+0000 7fe3cff52640 1 -- 172.21.15.117:0/1187508908 wait complete. 2024-05-15T13:20:25.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.907+0000 7fe3cff52640 10 monclient: build_initial_monmap 2024-05-15T13:20:25.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.907+0000 7fe3cff52640 10 monclient: monmap: 2024-05-15T13:20:25.909 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T13:20:25.909 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T13:20:25.910 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-05-15T13:20:25.910 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-05-15T13:20:25.910 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T13:20:25.910 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:25.910 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T13:20:25.910 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T13:20:25.910 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T13:20:25.910 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:25.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.908+0000 7fe3cff52640 1 librados: starting msgr at 2024-05-15T13:20:25.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.908+0000 7fe3cff52640 1 librados: starting objecter 2024-05-15T13:20:25.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.908+0000 7fe3cff52640 1 Processor -- start 2024-05-15T13:20:25.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.908+0000 7fe3cff52640 1 -- start start 2024-05-15T13:20:25.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.908+0000 7fe3cff52640 1 librados: setting wanted keys 2024-05-15T13:20:25.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.908+0000 7fe3cff52640 1 librados: calling monclient init 2024-05-15T13:20:25.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.908+0000 7fe3cff52640 10 monclient: init 2024-05-15T13:20:25.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.908+0000 7fe3cff52640 10 monclient: _reopen_session rank -1 2024-05-15T13:20:25.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.908+0000 7fe3cff52640 10 monclient: _add_conns ranks=[0,1,2] 2024-05-15T13:20:25.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.908+0000 7fe3cff52640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3c806bd70 0x7fe3c80a9e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:25.914 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.908+0000 7fe3cff52640 10 monclient(hunting): picked mon.noname-a con 0x7fe3c806bd70 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T13:20:25.914 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.908+0000 7fe3cff52640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe3c80aa3d0 0x7fe3c8134d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:25.914 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.908+0000 7fe3cff52640 10 monclient(hunting): picked mon.noname-b con 0x7fe3c80aa3d0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T13:20:25.914 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.908+0000 7fe3cff52640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe3c8135290 0x7fe3c81d6030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:25.914 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.908+0000 7fe3cff52640 10 monclient(hunting): picked mon.noname-c con 0x7fe3c8135290 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T13:20:25.914 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.908+0000 7fe3cff52640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:25.914 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.908+0000 7fe3cff52640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c80ae0b0 con 0x7fe3c806bd70 2024-05-15T13:20:25.914 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.908+0000 7fe3cff52640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:25.914 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.908+0000 7fe3cff52640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c80adf30 con 0x7fe3c80aa3d0 2024-05-15T13:20:25.914 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.908+0000 7fe3cff52640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:25.914 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.908+0000 7fe3cff52640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c80a9820 con 0x7fe3c8135290 2024-05-15T13:20:25.914 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.908+0000 7fe3cff52640 10 monclient(hunting): _renew_subs 2024-05-15T13:20:25.914 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.908+0000 7fe3cff52640 10 monclient(hunting): authenticate will time out at 3589.150770s 2024-05-15T13:20:25.914 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.909+0000 7fe3cdcee640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3c806bd70 0x7fe3c80a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:25.914 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.909+0000 7fe3ce4ef640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe3c8135290 0x7fe3c81d6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:25.914 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.909+0000 7fe3cd4ed640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe3c80aa3d0 0x7fe3c8134d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:25.915 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.909+0000 7fe3cdcee640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3c806bd70 0x7fe3c80a9e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:54756/0 (socket says 172.21.15.117:54756) 2024-05-15T13:20:25.915 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.909+0000 7fe3cdcee640 1 -- 172.21.15.117:0/1084866086 learned_addr learned my addr 172.21.15.117:0/1084866086 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T13:20:25.915 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.909+0000 7fe3cdcee640 10 monclient(hunting): get_auth_request con 0x7fe3c806bd70 auth_method 0 2024-05-15T13:20:25.915 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.909+0000 7fe3cdcee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:25.915 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.909+0000 7fe3cdcee640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:25.915 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.909+0000 7fe3cdcee640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:25.915 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.909+0000 7fe3cd4ed640 10 monclient(hunting): get_auth_request con 0x7fe3c80aa3d0 auth_method 0 2024-05-15T13:20:25.915 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.909+0000 7fe3cd4ed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:25.915 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.909+0000 7fe3cd4ed640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:25.915 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.909+0000 7fe3cd4ed640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:25.915 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.909+0000 7fe3ce4ef640 10 monclient(hunting): get_auth_request con 0x7fe3c8135290 auth_method 0 2024-05-15T13:20:25.915 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.909+0000 7fe3ce4ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:25.915 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.909+0000 7fe3ce4ef640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:25.915 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.909+0000 7fe3ce4ef640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:25.915 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.910+0000 7fe3cd4ed640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:25.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.910+0000 7fe3cd4ed640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:25.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.910+0000 7fe3cd4ed640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:25.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.910+0000 7fe3cdcee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:25.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.910+0000 7fe3cdcee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:25.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.910+0000 7fe3cdcee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:25.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.910+0000 7fe3ce4ef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:25.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.910+0000 7fe3ce4ef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:25.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.910+0000 7fe3ce4ef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:25.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.913+0000 7fe3cdcee640 10 monclient(hunting): handle_auth_done global_id 15993 payload 824 2024-05-15T13:20:25.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.913+0000 7fe3cdcee640 1 -- 172.21.15.117:0/1084866086 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe3c8135290 msgr2=0x7fe3c81d6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:25.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.913+0000 7fe3cdcee640 1 --2- 172.21.15.117:0/1084866086 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe3c8135290 0x7fe3c81d6030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:25.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.913+0000 7fe3cdcee640 1 -- 172.21.15.117:0/1084866086 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe3c80aa3d0 msgr2=0x7fe3c8134d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:25.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.913+0000 7fe3cdcee640 1 --2- 172.21.15.117:0/1084866086 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe3c80aa3d0 0x7fe3c8134d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:25.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.913+0000 7fe3cdcee640 10 monclient: _finish_hunting 0 2024-05-15T13:20:25.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.913+0000 7fe3cdcee640 1 monclient: found mon.noname-a 2024-05-15T13:20:25.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.913+0000 7fe3cdcee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:20:25.917 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.913+0000 7fe3cdcee640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.117:3300/0 2024-05-15T13:20:25.917 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.913+0000 7fe3cdcee640 1 -- 172.21.15.117:0/1084866086 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3c8135710 con 0x7fe3c806bd70 2024-05-15T13:20:25.917 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.913+0000 7fe3cd4ed640 1 --2- 172.21.15.117:0/1084866086 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe3c80aa3d0 0x7fe3c8134d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-15T13:20:25.917 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.913+0000 7fe3ce4ef640 1 --2- 172.21.15.117:0/1084866086 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe3c8135290 0x7fe3c81d6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-15T13:20:25.918 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.915+0000 7fe3cdcee640 1 --2- 172.21.15.117:0/1084866086 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3c806bd70 0x7fe3c80a9e90 secure :-1 s=READY pgs=3290 cs=0 l=1 rev1=1 crypto rx=0x7fe3c400e9e0 tx=0x7fe3c400eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:25.918 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.915+0000 7fe3be7fc640 10 client.?.objecter ms_handle_connect 0x7fe3c806bd70 2024-05-15T13:20:25.918 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.915+0000 7fe3be7fc640 10 client.?.objecter resend_mon_ops 2024-05-15T13:20:25.918 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.919+0000 7fe3be7fc640 1 -- 172.21.15.117:0/1084866086 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c4017070 con 0x7fe3c806bd70 2024-05-15T13:20:25.918 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.919+0000 7fe3be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:25.918 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.919+0000 7fe3be7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T13:20:25.918 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.919+0000 7fe3be7fc640 10 monclient: dump: 2024-05-15T13:20:25.918 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:25.919 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:25.919 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:25.919 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:25.919 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:25.919 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:25.919 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:25.919 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:25.919 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:25.919 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:25.919 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.919+0000 7fe3be7fc640 10 monclient: _finish_auth 0 2024-05-15T13:20:25.919 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.919+0000 7fe3be7fc640 10 monclient: _check_auth_tickets 2024-05-15T13:20:25.919 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.919+0000 7fe3be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:20:25.919 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.919+0000 7fe3be7fc640 1 -- 172.21.15.117:0/1084866086 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3c400c9d0 con 0x7fe3c806bd70 2024-05-15T13:20:25.919 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.919+0000 7fe3cff52640 5 monclient: authenticate success, global_id 15993 2024-05-15T13:20:25.919 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.919+0000 7fe3cff52640 10 monclient: _renew_subs 2024-05-15T13:20:25.919 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.919+0000 7fe3cff52640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T13:20:25.920 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.919+0000 7fe3cff52640 1 -- 172.21.15.117:0/1084866086 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3c80a94b0 con 0x7fe3c806bd70 2024-05-15T13:20:25.920 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.919+0000 7fe3be7fc640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:20:25.920 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.919+0000 7fe3be7fc640 1 -- 172.21.15.117:0/1084866086 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c400cdc0 con 0x7fe3c806bd70 2024-05-15T13:20:25.920 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.919+0000 7fe3be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:25.920 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.919+0000 7fe3be7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T13:20:25.920 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.919+0000 7fe3be7fc640 10 monclient: dump: 2024-05-15T13:20:25.920 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:25.920 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:25.920 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:25.920 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:25.920 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:25.920 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:25.920 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:25.920 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:25.920 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:25.920 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:25.921 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.919+0000 7fe3bffff640 4 set_mon_vals no callback set 2024-05-15T13:20:25.921 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.919+0000 7fe3cff52640 10 client.15993.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:20:25.921 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.919+0000 7fe3cff52640 10 monclient: _renew_subs 2024-05-15T13:20:25.921 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.919+0000 7fe3cff52640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T13:20:25.921 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.919+0000 7fe3cff52640 1 -- 172.21.15.117:0/1084866086 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3c8139d80 con 0x7fe3c806bd70 2024-05-15T13:20:25.921 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.919+0000 7fe3cff52640 1 librados: init done 2024-05-15T13:20:25.921 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.920+0000 7fe3cff52640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T13:20:25.922 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.920+0000 7fe3cff52640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:20:25.922 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.920+0000 7fe3cff52640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T13:20:25.922 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:25.920+0000 7fe3cff52640 1 -- 172.21.15.117:0/1084866086 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3c804f0e0 con 0x7fe3c806bd70 2024-05-15T13:20:26.167 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.168+0000 7fe3be7fc640 1 -- 172.21.15.117:0/1084866086 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fe3c401a070 con 0x7fe3c806bd70 2024-05-15T13:20:26.168 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.169+0000 7fe3be7fc640 1 --2- 172.21.15.117:0/1084866086 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fe3ac06ce90 0x7fe3ac06f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:26.168 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.169+0000 7fe3cd4ed640 1 --2- 172.21.15.117:0/1084866086 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fe3ac06ce90 0x7fe3ac06f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:26.168 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.169+0000 7fe3cd4ed640 10 monclient: get_auth_request con 0x7fe3ac06ce90 auth_method 0 2024-05-15T13:20:26.168 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.169+0000 7fe3be7fc640 10 client.15993.objecter ms_handle_connect 0x7fe3ac06ce90 2024-05-15T13:20:26.168 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.169+0000 7fe3cd4ed640 1 --2- 172.21.15.117:0/1084866086 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fe3ac06ce90 0x7fe3ac06f350 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fe3b8036000 tx=0x7fe3b8008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:26.181 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.183+0000 7fe3be7fc640 1 -- 172.21.15.117:0/1084866086 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(1224..1224 src has 912..1224) v4 ==== 6365+0+0 (secure 0 0 0) 0x7fe3c4056df0 con 0x7fe3c806bd70 2024-05-15T13:20:26.182 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.183+0000 7fe3be7fc640 10 client.15993.objecter ms_dispatch 0x7fe3c8000b90 osd_map(1224..1224 src has 912..1224) v4 2024-05-15T13:20:26.182 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.183+0000 7fe3be7fc640 3 client.15993.objecter handle_osd_map got epochs [1224,1224] > 0 2024-05-15T13:20:26.182 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.183+0000 7fe3be7fc640 3 client.15993.objecter handle_osd_map decoding full epoch 1224 2024-05-15T13:20:26.182 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.183+0000 7fe3be7fc640 20 client.15993.objecter dump_active .. 0 homeless 2024-05-15T13:20:26.182 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.183+0000 7fe3be7fc640 1 -- 172.21.15.117:0/1084866086 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe3c4058d50 con 0x7fe3c806bd70 2024-05-15T13:20:26.182 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.183+0000 7fe3be7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:20:26.182 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.183+0000 7fe3be7fc640 10 monclient: _finish_command 1 = system:0 2024-05-15T13:20:26.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.363+0000 7fe3cff52640 1 -- 172.21.15.117:0/1084866086 --> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe3c80ae230 con 0x7fe3ac06ce90 2024-05-15T13:20:26.364 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.366+0000 7fe3be7fc640 1 -- 172.21.15.117:0/1084866086 <== mgr.4107 v2:172.21.15.117:6800/1785766523 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+116276 (secure 0 0 0) 0x7fe3b8066650 con 0x7fe3ac06ce90 2024-05-15T13:20:26.365 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T13:20:26.366 INFO:teuthology.orchestra.run.smithi117.stderr:dumped all 2024-05-15T13:20:26.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.368+0000 7fe3cff52640 10 librados: watch_flush enter 2024-05-15T13:20:26.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.368+0000 7fe3cff52640 10 librados: watch_flush exit 2024-05-15T13:20:26.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.368+0000 7fe3cff52640 20 client.15993.objecter shutdown clearing up homeless session... 2024-05-15T13:20:26.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.368+0000 7fe3cff52640 10 client.15993.objecter successfully canceled tick 2024-05-15T13:20:26.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.368+0000 7fe3cff52640 1 -- 172.21.15.117:0/1084866086 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fe3ac06ce90 msgr2=0x7fe3ac06f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:26.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.368+0000 7fe3cff52640 1 --2- 172.21.15.117:0/1084866086 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fe3ac06ce90 0x7fe3ac06f350 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fe3b8036000 tx=0x7fe3b8008030 comp rx=0 tx=0).stop 2024-05-15T13:20:26.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.368+0000 7fe3cff52640 10 monclient: shutdown 2024-05-15T13:20:26.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.368+0000 7fe3cff52640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:20:26.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.368+0000 7fe3cff52640 1 -- 172.21.15.117:0/1084866086 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3c806bd70 msgr2=0x7fe3c80a9e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:26.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.368+0000 7fe3cff52640 1 --2- 172.21.15.117:0/1084866086 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3c806bd70 0x7fe3c80a9e90 secure :-1 s=READY pgs=3290 cs=0 l=1 rev1=1 crypto rx=0x7fe3c400e9e0 tx=0x7fe3c400eea0 comp rx=0 tx=0).stop 2024-05-15T13:20:26.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.368+0000 7fe3cff52640 1 -- 172.21.15.117:0/1084866086 shutdown_connections 2024-05-15T13:20:26.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.368+0000 7fe3cff52640 1 --2- 172.21.15.117:0/1084866086 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe3c8135290 0x7fe3c81d6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:26.369 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.368+0000 7fe3cff52640 1 --2- 172.21.15.117:0/1084866086 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fe3ac06ce90 0x7fe3ac06f350 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:26.369 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.368+0000 7fe3cff52640 1 --2- 172.21.15.117:0/1084866086 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe3c80aa3d0 0x7fe3c8134d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:26.369 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.368+0000 7fe3cff52640 1 --2- 172.21.15.117:0/1084866086 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3c806bd70 0x7fe3c80a9e90 unknown :-1 s=CLOSED pgs=3290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:26.369 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.368+0000 7fe3cff52640 1 -- 172.21.15.117:0/1084866086 >> 172.21.15.117:0/1084866086 conn(0x7fe3c806ada0 msgr2=0x7fe3c80af3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:20:26.369 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.368+0000 7fe3cff52640 1 -- 172.21.15.117:0/1084866086 shutdown_connections 2024-05-15T13:20:26.369 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.368+0000 7fe3cff52640 1 -- 172.21.15.117:0/1084866086 wait complete. 2024-05-15T13:20:26.369 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.368+0000 7fe3cff52640 1 librados: shutdown 2024-05-15T13:20:26.378 INFO:teuthology.orchestra.run.smithi117.stdout:{"pg_ready":true,"pg_map":{"version":2355,"stamp":"2024-05-15T13:20:24.612592+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":85,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":602048,"kb_used_data":41680,"kb_used_omap":12,"kb_used_meta":560243,"kb_avail":754372672,"statfs":{"total":773094113280,"available":772477616128,"internally_reserved":0,"allocated":42680320,"data_stored":21281579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":573689394},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":141,"apply_latency_ms":141,"commit_latency_ns":141000000,"apply_latency_ns":141000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.009369"},"pg_stats":[{"pgid":"6.16","version":"0'0","reported_seq":2354,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.597416+0000","last_change":"2024-05-15T12:44:59.567389+0000","last_active":"2024-05-15T13:20:19.597416+0000","last_peered":"2024-05-15T13:20:19.597416+0000","last_clean":"2024-05-15T13:20:19.597416+0000","last_became_active":"2024-05-15T12:44:59.550282+0000","last_became_peered":"2024-05-15T12:44:59.550282+0000","last_unstale":"2024-05-15T13:20:19.597416+0000","last_undegraded":"2024-05-15T13:20:19.597416+0000","last_fullsized":"2024-05-15T13:20:19.597416+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:44:22.143842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":2369,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.612783+0000","last_change":"2024-05-15T12:44:51.243387+0000","last_active":"2024-05-15T13:20:19.612783+0000","last_peered":"2024-05-15T13:20:19.612783+0000","last_clean":"2024-05-15T13:20:19.612783+0000","last_became_active":"2024-05-15T12:44:51.202412+0000","last_became_peered":"2024-05-15T12:44:51.202412+0000","last_unstale":"2024-05-15T13:20:19.612783+0000","last_undegraded":"2024-05-15T13:20:19.612783+0000","last_fullsized":"2024-05-15T13:20:19.612783+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:46:23.223713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":2355,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.590005+0000","last_change":"2024-05-15T12:45:13.532350+0000","last_active":"2024-05-15T13:20:19.590005+0000","last_peered":"2024-05-15T13:20:19.590005+0000","last_clean":"2024-05-15T13:20:19.590005+0000","last_became_active":"2024-05-15T12:45:11.468881+0000","last_became_peered":"2024-05-15T12:45:11.468881+0000","last_unstale":"2024-05-15T13:20:19.590005+0000","last_undegraded":"2024-05-15T13:20:19.590005+0000","last_fullsized":"2024-05-15T13:20:19.590005+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T15:53:17.739425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":2357,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.512008+0000","last_change":"2024-05-15T12:44:55.797152+0000","last_active":"2024-05-15T13:20:19.512008+0000","last_peered":"2024-05-15T13:20:19.512008+0000","last_clean":"2024-05-15T13:20:19.512008+0000","last_became_active":"2024-05-15T12:44:55.775514+0000","last_became_peered":"2024-05-15T12:44:55.775514+0000","last_unstale":"2024-05-15T13:20:19.512008+0000","last_undegraded":"2024-05-15T13:20:19.512008+0000","last_fullsized":"2024-05-15T13:20:19.512008+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:09:17.199545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":2355,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.614535+0000","last_change":"2024-05-15T12:44:57.700224+0000","last_active":"2024-05-15T13:20:19.614535+0000","last_peered":"2024-05-15T13:20:19.614535+0000","last_clean":"2024-05-15T13:20:19.614535+0000","last_became_active":"2024-05-15T12:44:57.673614+0000","last_became_peered":"2024-05-15T12:44:57.673614+0000","last_unstale":"2024-05-15T13:20:19.614535+0000","last_undegraded":"2024-05-15T13:20:19.614535+0000","last_fullsized":"2024-05-15T13:20:19.614535+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-17T00:09:31.317495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":2355,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.583562+0000","last_change":"2024-05-15T12:44:59.486234+0000","last_active":"2024-05-15T13:20:19.583562+0000","last_peered":"2024-05-15T13:20:19.583562+0000","last_clean":"2024-05-15T13:20:19.583562+0000","last_became_active":"2024-05-15T12:44:59.480451+0000","last_became_peered":"2024-05-15T12:44:59.480451+0000","last_unstale":"2024-05-15T13:20:19.583562+0000","last_undegraded":"2024-05-15T13:20:19.583562+0000","last_fullsized":"2024-05-15T13:20:19.583562+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:28:22.234323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":2361,"reported_epoch":1221,"state":"active+clean","last_fresh":"2024-05-15T13:20:20.722753+0000","last_change":"2024-05-15T12:44:54.924153+0000","last_active":"2024-05-15T13:20:20.722753+0000","last_peered":"2024-05-15T13:20:20.722753+0000","last_clean":"2024-05-15T13:20:20.722753+0000","last_became_active":"2024-05-15T12:44:54.914910+0000","last_became_peered":"2024-05-15T12:44:54.914910+0000","last_unstale":"2024-05-15T13:20:20.722753+0000","last_undegraded":"2024-05-15T13:20:20.722753+0000","last_fullsized":"2024-05-15T13:20:20.722753+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-17T00:36:49.391069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":2356,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.564063+0000","last_change":"2024-05-15T12:44:59.650327+0000","last_active":"2024-05-15T13:20:19.564063+0000","last_peered":"2024-05-15T13:20:19.564063+0000","last_clean":"2024-05-15T13:20:19.564063+0000","last_became_active":"2024-05-15T12:44:59.595062+0000","last_became_peered":"2024-05-15T12:44:59.595062+0000","last_unstale":"2024-05-15T13:20:19.564063+0000","last_undegraded":"2024-05-15T13:20:19.564063+0000","last_fullsized":"2024-05-15T13:20:19.564063+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T12:52:31.383828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":2371,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.607147+0000","last_change":"2024-05-15T12:44:51.802252+0000","last_active":"2024-05-15T13:20:19.607147+0000","last_peered":"2024-05-15T13:20:19.607147+0000","last_clean":"2024-05-15T13:20:19.607147+0000","last_became_active":"2024-05-15T12:44:51.785998+0000","last_became_peered":"2024-05-15T12:44:51.785998+0000","last_unstale":"2024-05-15T13:20:19.607147+0000","last_undegraded":"2024-05-15T13:20:19.607147+0000","last_fullsized":"2024-05-15T13:20:19.607147+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:00:30.969147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":2372,"reported_epoch":1221,"state":"active+clean","last_fresh":"2024-05-15T13:20:20.670827+0000","last_change":"2024-05-15T12:44:53.614201+0000","last_active":"2024-05-15T13:20:20.670827+0000","last_peered":"2024-05-15T13:20:20.670827+0000","last_clean":"2024-05-15T13:20:20.670827+0000","last_became_active":"2024-05-15T12:44:53.608407+0000","last_became_peered":"2024-05-15T12:44:53.608407+0000","last_unstale":"2024-05-15T13:20:20.670827+0000","last_undegraded":"2024-05-15T13:20:20.670827+0000","last_fullsized":"2024-05-15T13:20:20.670827+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:45:19.480038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":2362,"reported_epoch":1222,"state":"active+clean","last_fresh":"2024-05-15T13:20:22.298535+0000","last_change":"2024-05-15T12:44:52.221672+0000","last_active":"2024-05-15T13:20:22.298535+0000","last_peered":"2024-05-15T13:20:22.298535+0000","last_clean":"2024-05-15T13:20:22.298535+0000","last_became_active":"2024-05-15T12:44:52.212480+0000","last_became_peered":"2024-05-15T12:44:52.212480+0000","last_unstale":"2024-05-15T13:20:22.298535+0000","last_undegraded":"2024-05-15T13:20:22.298535+0000","last_fullsized":"2024-05-15T13:20:22.298535+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:41:34.164739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":2371,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.609138+0000","last_change":"2024-05-15T12:44:56.446419+0000","last_active":"2024-05-15T13:20:19.609138+0000","last_peered":"2024-05-15T13:20:19.609138+0000","last_clean":"2024-05-15T13:20:19.609138+0000","last_became_active":"2024-05-15T12:44:56.197837+0000","last_became_peered":"2024-05-15T12:44:56.197837+0000","last_unstale":"2024-05-15T13:20:19.609138+0000","last_undegraded":"2024-05-15T13:20:19.609138+0000","last_fullsized":"2024-05-15T13:20:19.609138+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:33:22.475565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":2360,"reported_epoch":1221,"state":"active+clean","last_fresh":"2024-05-15T13:20:20.814121+0000","last_change":"2024-05-15T12:44:53.399230+0000","last_active":"2024-05-15T13:20:20.814121+0000","last_peered":"2024-05-15T13:20:20.814121+0000","last_clean":"2024-05-15T13:20:20.814121+0000","last_became_active":"2024-05-15T12:44:53.377773+0000","last_became_peered":"2024-05-15T12:44:53.377773+0000","last_unstale":"2024-05-15T13:20:20.814121+0000","last_undegraded":"2024-05-15T13:20:20.814121+0000","last_fullsized":"2024-05-15T13:20:20.814121+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:12:39.424987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'76","reported_seq":2393,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.589082+0000","last_change":"2024-05-15T12:43:04.763767+0000","last_active":"2024-05-15T13:20:19.589082+0000","last_peered":"2024-05-15T13:20:19.589082+0000","last_clean":"2024-05-15T13:20:19.589082+0000","last_became_active":"2024-05-15T12:43:04.148735+0000","last_became_peered":"2024-05-15T12:43:04.148735+0000","last_unstale":"2024-05-15T13:20:19.589082+0000","last_undegraded":"2024-05-15T13:20:19.589082+0000","last_fullsized":"2024-05-15T13:20:19.589082+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:42:53.071197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:42:53.071197+0000","last_clean_scrub_stamp":"2024-05-15T12:42:53.071197+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:45:23.691461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2370,"reported_epoch":1221,"state":"active+clean","last_fresh":"2024-05-15T13:20:20.778978+0000","last_change":"2024-05-15T12:43:09.453018+0000","last_active":"2024-05-15T13:20:20.778978+0000","last_peered":"2024-05-15T13:20:20.778978+0000","last_clean":"2024-05-15T13:20:20.778978+0000","last_became_active":"2024-05-15T12:43:09.445878+0000","last_became_peered":"2024-05-15T12:43:09.445878+0000","last_unstale":"2024-05-15T13:20:20.778978+0000","last_undegraded":"2024-05-15T13:20:20.778978+0000","last_fullsized":"2024-05-15T13:20:20.778978+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_clean_scrub_stamp":"2024-05-15T12:43:07.133352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:55:05.766298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":2364,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.629451+0000","last_change":"2024-05-15T12:44:50.705660+0000","last_active":"2024-05-15T13:20:19.629451+0000","last_peered":"2024-05-15T13:20:19.629451+0000","last_clean":"2024-05-15T13:20:19.629451+0000","last_became_active":"2024-05-15T12:44:50.696664+0000","last_became_peered":"2024-05-15T12:44:50.696664+0000","last_unstale":"2024-05-15T13:20:19.629451+0000","last_undegraded":"2024-05-15T13:20:19.629451+0000","last_fullsized":"2024-05-15T13:20:19.629451+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:15:31.819701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2371,"reported_epoch":1221,"state":"active+clean","last_fresh":"2024-05-15T13:20:20.720837+0000","last_change":"2024-05-15T12:43:08.704425+0000","last_active":"2024-05-15T13:20:20.720837+0000","last_peered":"2024-05-15T13:20:20.720837+0000","last_clean":"2024-05-15T13:20:20.720837+0000","last_became_active":"2024-05-15T12:43:08.677751+0000","last_became_peered":"2024-05-15T12:43:08.677751+0000","last_unstale":"2024-05-15T13:20:20.720837+0000","last_undegraded":"2024-05-15T13:20:20.720837+0000","last_fullsized":"2024-05-15T13:20:20.720837+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_clean_scrub_stamp":"2024-05-15T12:43:07.133352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-17T00:22:51.843061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":2355,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.638687+0000","last_change":"2024-05-15T12:44:57.579283+0000","last_active":"2024-05-15T13:20:19.638687+0000","last_peered":"2024-05-15T13:20:19.638687+0000","last_clean":"2024-05-15T13:20:19.638687+0000","last_became_active":"2024-05-15T12:44:57.552111+0000","last_became_peered":"2024-05-15T12:44:57.552111+0000","last_unstale":"2024-05-15T13:20:19.638687+0000","last_undegraded":"2024-05-15T13:20:19.638687+0000","last_fullsized":"2024-05-15T13:20:19.638687+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:23:14.388995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2372,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.618288+0000","last_change":"2024-05-15T12:43:08.634999+0000","last_active":"2024-05-15T13:20:19.618288+0000","last_peered":"2024-05-15T13:20:19.618288+0000","last_clean":"2024-05-15T13:20:19.618288+0000","last_became_active":"2024-05-15T12:43:08.600745+0000","last_became_peered":"2024-05-15T12:43:08.600745+0000","last_unstale":"2024-05-15T13:20:19.618288+0000","last_undegraded":"2024-05-15T13:20:19.618288+0000","last_fullsized":"2024-05-15T13:20:19.618288+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_clean_scrub_stamp":"2024-05-15T12:43:07.133352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:17:56.643544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":2373,"reported_epoch":1221,"state":"active+clean","last_fresh":"2024-05-15T13:20:20.656371+0000","last_change":"2024-05-15T12:44:51.361084+0000","last_active":"2024-05-15T13:20:20.656371+0000","last_peered":"2024-05-15T13:20:20.656371+0000","last_clean":"2024-05-15T13:20:20.656371+0000","last_became_active":"2024-05-15T12:44:51.354879+0000","last_became_peered":"2024-05-15T12:44:51.354879+0000","last_unstale":"2024-05-15T13:20:20.656371+0000","last_undegraded":"2024-05-15T13:20:20.656371+0000","last_fullsized":"2024-05-15T13:20:20.656371+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:53:33.707185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2370,"reported_epoch":1221,"state":"active+clean","last_fresh":"2024-05-15T13:20:20.765250+0000","last_change":"2024-05-15T12:43:09.375465+0000","last_active":"2024-05-15T13:20:20.765250+0000","last_peered":"2024-05-15T13:20:20.765250+0000","last_clean":"2024-05-15T13:20:20.765250+0000","last_became_active":"2024-05-15T12:43:09.363876+0000","last_became_peered":"2024-05-15T12:43:09.363876+0000","last_unstale":"2024-05-15T13:20:20.765250+0000","last_undegraded":"2024-05-15T13:20:20.765250+0000","last_fullsized":"2024-05-15T13:20:20.765250+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_clean_scrub_stamp":"2024-05-15T12:43:07.133352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:27:46.599385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":2364,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.594769+0000","last_change":"2024-05-15T12:44:54.188435+0000","last_active":"2024-05-15T13:20:19.594769+0000","last_peered":"2024-05-15T13:20:19.594769+0000","last_clean":"2024-05-15T13:20:19.594769+0000","last_became_active":"2024-05-15T12:44:54.179460+0000","last_became_peered":"2024-05-15T12:44:54.179460+0000","last_unstale":"2024-05-15T13:20:19.594769+0000","last_undegraded":"2024-05-15T13:20:19.594769+0000","last_fullsized":"2024-05-15T13:20:19.594769+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T12:52:27.875820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2382,"reported_epoch":1221,"state":"active+clean","last_fresh":"2024-05-15T13:20:20.653740+0000","last_change":"2024-05-15T12:43:09.384765+0000","last_active":"2024-05-15T13:20:20.653740+0000","last_peered":"2024-05-15T13:20:20.653740+0000","last_clean":"2024-05-15T13:20:20.653740+0000","last_became_active":"2024-05-15T12:43:09.377725+0000","last_became_peered":"2024-05-15T12:43:09.377725+0000","last_unstale":"2024-05-15T13:20:20.653740+0000","last_undegraded":"2024-05-15T13:20:20.653740+0000","last_fullsized":"2024-05-15T13:20:20.653740+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_clean_scrub_stamp":"2024-05-15T12:43:07.133352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:57:59.671602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"47'7","reported_seq":2377,"reported_epoch":1222,"state":"active+clean","last_fresh":"2024-05-15T13:20:22.279258+0000","last_change":"2024-05-15T12:44:58.436179+0000","last_active":"2024-05-15T13:20:22.279258+0000","last_peered":"2024-05-15T13:20:22.279258+0000","last_clean":"2024-05-15T13:20:22.279258+0000","last_became_active":"2024-05-15T12:44:58.318306+0000","last_became_peered":"2024-05-15T12:44:58.318306+0000","last_unstale":"2024-05-15T13:20:22.279258+0000","last_undegraded":"2024-05-15T13:20:22.279258+0000","last_fullsized":"2024-05-15T13:20:22.279258+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:59:46.310559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2370,"reported_epoch":1221,"state":"active+clean","last_fresh":"2024-05-15T13:20:20.794871+0000","last_change":"2024-05-15T12:43:08.778833+0000","last_active":"2024-05-15T13:20:20.794871+0000","last_peered":"2024-05-15T13:20:20.794871+0000","last_clean":"2024-05-15T13:20:20.794871+0000","last_became_active":"2024-05-15T12:43:08.773614+0000","last_became_peered":"2024-05-15T12:43:08.773614+0000","last_unstale":"2024-05-15T13:20:20.794871+0000","last_undegraded":"2024-05-15T13:20:20.794871+0000","last_fullsized":"2024-05-15T13:20:20.794871+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_clean_scrub_stamp":"2024-05-15T12:43:07.133352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:22:59.785687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":2361,"reported_epoch":1221,"state":"active+clean","last_fresh":"2024-05-15T13:20:20.771718+0000","last_change":"2024-05-15T12:45:08.669407+0000","last_active":"2024-05-15T13:20:20.771718+0000","last_peered":"2024-05-15T13:20:20.771718+0000","last_clean":"2024-05-15T13:20:20.771718+0000","last_became_active":"2024-05-15T12:45:08.661140+0000","last_became_peered":"2024-05-15T12:45:08.661140+0000","last_unstale":"2024-05-15T13:20:20.771718+0000","last_undegraded":"2024-05-15T13:20:20.771718+0000","last_fullsized":"2024-05-15T13:20:20.771718+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:55:43.700115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2366,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.599275+0000","last_change":"2024-05-15T12:43:09.478046+0000","last_active":"2024-05-15T13:20:19.599275+0000","last_peered":"2024-05-15T13:20:19.599275+0000","last_clean":"2024-05-15T13:20:19.599275+0000","last_became_active":"2024-05-15T12:43:09.448237+0000","last_became_peered":"2024-05-15T12:43:09.448237+0000","last_unstale":"2024-05-15T13:20:19.599275+0000","last_undegraded":"2024-05-15T13:20:19.599275+0000","last_fullsized":"2024-05-15T13:20:19.599275+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_clean_scrub_stamp":"2024-05-15T12:43:07.133352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:11:17.055288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":2355,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.612460+0000","last_change":"2024-05-15T12:45:18.657708+0000","last_active":"2024-05-15T13:20:19.612460+0000","last_peered":"2024-05-15T13:20:19.612460+0000","last_clean":"2024-05-15T13:20:19.612460+0000","last_became_active":"2024-05-15T12:45:18.642568+0000","last_became_peered":"2024-05-15T12:45:18.642568+0000","last_unstale":"2024-05-15T13:20:19.612460+0000","last_undegraded":"2024-05-15T13:20:19.612460+0000","last_fullsized":"2024-05-15T13:20:19.612460+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:21:30.095711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2369,"reported_epoch":1221,"state":"active+clean","last_fresh":"2024-05-15T13:20:20.815988+0000","last_change":"2024-05-15T12:43:09.429828+0000","last_active":"2024-05-15T13:20:20.815988+0000","last_peered":"2024-05-15T13:20:20.815988+0000","last_clean":"2024-05-15T13:20:20.815988+0000","last_became_active":"2024-05-15T12:43:09.424460+0000","last_became_peered":"2024-05-15T12:43:09.424460+0000","last_unstale":"2024-05-15T13:20:20.815988+0000","last_undegraded":"2024-05-15T13:20:20.815988+0000","last_fullsized":"2024-05-15T13:20:20.815988+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_clean_scrub_stamp":"2024-05-15T12:43:07.133352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:12:21.424590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":2362,"reported_epoch":1222,"state":"active+clean","last_fresh":"2024-05-15T13:20:22.277210+0000","last_change":"2024-05-15T12:44:52.129096+0000","last_active":"2024-05-15T13:20:22.277210+0000","last_peered":"2024-05-15T13:20:22.277210+0000","last_clean":"2024-05-15T13:20:22.277210+0000","last_became_active":"2024-05-15T12:44:52.088745+0000","last_became_peered":"2024-05-15T12:44:52.088745+0000","last_unstale":"2024-05-15T13:20:22.277210+0000","last_undegraded":"2024-05-15T13:20:22.277210+0000","last_fullsized":"2024-05-15T13:20:22.277210+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:16:34.966002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":2355,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.552600+0000","last_change":"2024-05-15T12:44:56.480904+0000","last_active":"2024-05-15T13:20:19.552600+0000","last_peered":"2024-05-15T13:20:19.552600+0000","last_clean":"2024-05-15T13:20:19.552600+0000","last_became_active":"2024-05-15T12:44:56.470246+0000","last_became_peered":"2024-05-15T12:44:56.470246+0000","last_unstale":"2024-05-15T13:20:19.552600+0000","last_undegraded":"2024-05-15T13:20:19.552600+0000","last_fullsized":"2024-05-15T13:20:19.552600+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:53:59.488101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":2361,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.610996+0000","last_change":"2024-05-15T12:44:51.677390+0000","last_active":"2024-05-15T13:20:19.610996+0000","last_peered":"2024-05-15T13:20:19.610996+0000","last_clean":"2024-05-15T13:20:19.610996+0000","last_became_active":"2024-05-15T12:44:51.660527+0000","last_became_peered":"2024-05-15T12:44:51.660527+0000","last_unstale":"2024-05-15T13:20:19.610996+0000","last_undegraded":"2024-05-15T13:20:19.610996+0000","last_fullsized":"2024-05-15T13:20:19.610996+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:25:49.920807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":2354,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.616476+0000","last_change":"2024-05-15T12:45:12.010393+0000","last_active":"2024-05-15T13:20:19.616476+0000","last_peered":"2024-05-15T13:20:19.616476+0000","last_clean":"2024-05-15T13:20:19.616476+0000","last_became_active":"2024-05-15T12:45:11.998256+0000","last_became_peered":"2024-05-15T12:45:11.998256+0000","last_unstale":"2024-05-15T13:20:19.616476+0000","last_undegraded":"2024-05-15T13:20:19.616476+0000","last_fullsized":"2024-05-15T13:20:19.616476+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:13:05.556724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":2360,"reported_epoch":1222,"state":"active+clean","last_fresh":"2024-05-15T13:20:22.317768+0000","last_change":"2024-05-15T12:44:55.564951+0000","last_active":"2024-05-15T13:20:22.317768+0000","last_peered":"2024-05-15T13:20:22.317768+0000","last_clean":"2024-05-15T13:20:22.317768+0000","last_became_active":"2024-05-15T12:44:55.557214+0000","last_became_peered":"2024-05-15T12:44:55.557214+0000","last_unstale":"2024-05-15T13:20:22.317768+0000","last_undegraded":"2024-05-15T13:20:22.317768+0000","last_fullsized":"2024-05-15T13:20:22.317768+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:53:15.834038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":2359,"reported_epoch":1221,"state":"active+clean","last_fresh":"2024-05-15T13:20:20.785961+0000","last_change":"2024-05-15T12:44:56.078343+0000","last_active":"2024-05-15T13:20:20.785961+0000","last_peered":"2024-05-15T13:20:20.785961+0000","last_clean":"2024-05-15T13:20:20.785961+0000","last_became_active":"2024-05-15T12:44:56.072618+0000","last_became_peered":"2024-05-15T12:44:56.072618+0000","last_unstale":"2024-05-15T13:20:20.785961+0000","last_undegraded":"2024-05-15T13:20:20.785961+0000","last_fullsized":"2024-05-15T13:20:20.785961+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:22:00.980595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":2364,"reported_epoch":1222,"state":"active+clean","last_fresh":"2024-05-15T13:20:22.332975+0000","last_change":"2024-05-15T12:44:55.746501+0000","last_active":"2024-05-15T13:20:22.332975+0000","last_peered":"2024-05-15T13:20:22.332975+0000","last_clean":"2024-05-15T13:20:22.332975+0000","last_became_active":"2024-05-15T12:44:55.739811+0000","last_became_peered":"2024-05-15T12:44:55.739811+0000","last_unstale":"2024-05-15T13:20:22.332975+0000","last_undegraded":"2024-05-15T13:20:22.332975+0000","last_fullsized":"2024-05-15T13:20:22.332975+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T12:49:40.051745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":2361,"reported_epoch":1221,"state":"active+clean","last_fresh":"2024-05-15T13:20:20.800642+0000","last_change":"2024-05-15T12:44:56.495925+0000","last_active":"2024-05-15T13:20:20.800642+0000","last_peered":"2024-05-15T13:20:20.800642+0000","last_clean":"2024-05-15T13:20:20.800642+0000","last_became_active":"2024-05-15T12:44:56.429996+0000","last_became_peered":"2024-05-15T12:44:56.429996+0000","last_unstale":"2024-05-15T13:20:20.800642+0000","last_undegraded":"2024-05-15T13:20:20.800642+0000","last_fullsized":"2024-05-15T13:20:20.800642+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:18:31.842665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":2353,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.604742+0000","last_change":"2024-05-15T12:45:18.556824+0000","last_active":"2024-05-15T13:20:19.604742+0000","last_peered":"2024-05-15T13:20:19.604742+0000","last_clean":"2024-05-15T13:20:19.604742+0000","last_became_active":"2024-05-15T12:45:18.546529+0000","last_became_peered":"2024-05-15T12:45:18.546529+0000","last_unstale":"2024-05-15T13:20:19.604742+0000","last_undegraded":"2024-05-15T13:20:19.604742+0000","last_fullsized":"2024-05-15T13:20:19.604742+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:07:19.212044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":2354,"reported_epoch":1220,"state":"active+clean","last_fresh":"2024-05-15T13:20:19.555166+0000","last_change":"2024-05-15T12:44:56.642476+0000","last_active":"2024-05-15T13:20:19.555166+0000","last_peered":"2024-05-15T13:20:19.555166+0000","last_clean":"2024-05-15T13:20:19.555166+0000","last_became_active":"2024-05-15T12:44:56.634257+0000","last_became_peered":"2024-05-15T12:44:56.634257+0000","last_unstale":"2024-05-15T13:20:19.555166+0000","last_undegraded":"2024-05-15T13:20:19.555166+0000","last_fullsized":"2024-05-15T13:20:19.555166+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T15:44:34.817095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":2361,"reported_epoch":1222,"state":"active+clean","last_fresh":"2024-05-15T13:20:22.287049+0000","last_change":"2024-05-15T12:44:52.504602+0000","last_active":"2024-05-15T13:20:22.287049+0000","last_peered":"2024-05-15T13:20:22.287049+0000","last_clean":"2024-05-15T13:20:22.287049+0000","last_became_active":"2024-05-15T12:44:52.483168+0000","last_became_peered":"2024-05-15T12:44:52.483168+0000","last_unstale":"2024-05-15T13:20:22.287049+0000","last_undegraded":"2024-05-15T13:20:22.287049+0000","last_fullsized":"2024-05-15T13:20:22.287049+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:17:24.835749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":2359,"reported_epoch":1221,"state":"active+clean","last_fresh":"2024-05-15T13:20:20.768531+0000","last_change":"2024-05-15T12:45:12.195717+0000","last_active":"2024-05-15T13:20:20.768531+0000","last_peered":"2024-05-15T13:20:20.768531+0000","last_clean":"2024-05-15T13:20:20.768531+0000","last_became_active":"2024-05-15T12:45:12.190202+0000","last_became_peered":"2024-05-15T12:45:12.190202+0000","last_unstale":"2024-05-15T13:20:20.768531+0000","last_undegraded":"2024-05-15T13:20:20.768531+0000","last_fullsized":"2024-05-15T13:20:20.768531+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:05:34.894666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":20,"seq":85899346373,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72784,"kb_used_data":5504,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299056,"statfs":{"total":96636764160,"available":96562233344,"internally_reserved":0,"allocated":5636096,"data_stored":2959141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[]},{"osd":0,"up_from":20,"seq":85899346372,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68220,"kb_used_data":5100,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303620,"statfs":{"total":96636764160,"available":96566906880,"internally_reserved":0,"allocated":5222400,"data_stored":2553456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[]},{"osd":2,"up_from":20,"seq":85899346372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94532,"kb_used_data":5112,"kb_used_omap":1,"kb_used_meta":89406,"kb_avail":94277308,"statfs":{"total":96636764160,"available":96539963392,"internally_reserved":0,"allocated":5234688,"data_stored":2561301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":91552198},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":6,"up_from":19,"seq":81604379076,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72404,"kb_used_data":5128,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299436,"statfs":{"total":96636764160,"available":96562622464,"internally_reserved":0,"allocated":5251072,"data_stored":2568766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[]},{"osd":4,"up_from":19,"seq":81604379076,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72376,"kb_used_data":5100,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299464,"statfs":{"total":96636764160,"available":96562651136,"internally_reserved":0,"allocated":5222400,"data_stored":2553456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509890,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72396,"kb_used_data":5116,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299444,"statfs":{"total":96636764160,"available":96562630656,"internally_reserved":0,"allocated":5238784,"data_stored":2561306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542595,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72796,"kb_used_data":5520,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299044,"statfs":{"total":96636764160,"available":96562221056,"internally_reserved":0,"allocated":5652480,"data_stored":2970697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[]},{"osd":7,"up_from":13,"seq":55834575300,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76540,"kb_used_data":5100,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94295300,"statfs":{"total":96636764160,"available":96558387200,"internally_reserved":0,"allocated":5222400,"data_stored":2553456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T13:20:26.380 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-15T13:20:26.480 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.481+0000 7fdb1ad79640 10 monclient: get_monmap_and_config 2024-05-15T13:20:26.480 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.481+0000 7fdb1ad79640 10 monclient: build_initial_monmap 2024-05-15T13:20:26.480 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.481+0000 7fdb1ad79640 10 monclient: monmap: 2024-05-15T13:20:26.480 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T13:20:26.480 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:26.480 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T13:20:26.482560+0000 2024-05-15T13:20:26.480 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T13:20:26.482560+0000 2024-05-15T13:20:26.480 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T13:20:26.480 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:26.481 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T13:20:26.481 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T13:20:26.481 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T13:20:26.481 INFO:teuthology.orchestra.run.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T13:20:26.481 INFO:teuthology.orchestra.run.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T13:20:26.481 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:26.483 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.482+0000 7fdb1ad79640 1 Processor -- start 2024-05-15T13:20:26.483 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.482+0000 7fdb1ad79640 1 -- start start 2024-05-15T13:20:26.483 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.482+0000 7fdb1ad79640 10 monclient: init 2024-05-15T13:20:26.483 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.482+0000 7fdb1ad79640 10 monclient: _reopen_session rank -1 2024-05-15T13:20:26.483 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.482+0000 7fdb1ad79640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-05-15T13:20:26.483 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.482+0000 7fdb1ad79640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fdb1406b010 0x7fdb14143a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:26.483 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.482+0000 7fdb1ad79640 10 monclient(hunting): picked mon.noname-a con 0x7fdb1406b010 addr v2:172.21.15.117:3300/0 2024-05-15T13:20:26.484 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.482+0000 7fdb1ad79640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdb140b30c0 addr v1:172.21.15.117:6789/0 2024-05-15T13:20:26.484 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.482+0000 7fdb1ad79640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fdb14143f80 0x7fdb14071a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:26.484 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.482+0000 7fdb1ad79640 10 monclient(hunting): picked mon.noname-c con 0x7fdb14143f80 addr v2:172.21.15.178:3300/0 2024-05-15T13:20:26.484 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.482+0000 7fdb1ad79640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb140b4740 con 0x7fdb140b30c0 2024-05-15T13:20:26.484 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.482+0000 7fdb1ad79640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:26.484 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.482+0000 7fdb1ad79640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb14144430 con 0x7fdb1406b010 2024-05-15T13:20:26.484 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.482+0000 7fdb1ad79640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:26.484 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.482+0000 7fdb1ad79640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb14072f00 con 0x7fdb14143f80 2024-05-15T13:20:26.484 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.482+0000 7fdb1ad79640 10 monclient(hunting): _renew_subs 2024-05-15T13:20:26.484 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.482+0000 7fdb1ad79640 10 monclient(hunting): authenticate will time out at 3589.724734s 2024-05-15T13:20:26.484 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.483+0000 7fdb19316640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fdb14143f80 0x7fdb14071a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:26.484 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.483+0000 7fdb18b15640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fdb1406b010 0x7fdb14143a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:26.484 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.483+0000 7fdb13fff640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fdb140b30c0 0x7fdb14071320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:54080/0 (socket says 172.21.15.117:54080) 2024-05-15T13:20:26.484 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.483+0000 7fdb13fff640 1 -- 172.21.15.117:0/1730012654 learned_addr learned my addr 172.21.15.117:0/1730012654 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-05-15T13:20:26.484 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.483+0000 7fdb19316640 10 monclient(hunting): get_auth_request con 0x7fdb14143f80 auth_method 0 2024-05-15T13:20:26.484 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.483+0000 7fdb19316640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:26.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.483+0000 7fdb19316640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:26.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.483+0000 7fdb19316640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:26.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.484+0000 7fdb18b15640 10 monclient(hunting): get_auth_request con 0x7fdb1406b010 auth_method 0 2024-05-15T13:20:26.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.484+0000 7fdb18b15640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:26.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.484+0000 7fdb18b15640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:26.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.484+0000 7fdb18b15640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:26.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.484+0000 7fdb19316640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:26.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.484+0000 7fdb19316640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:26.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.484+0000 7fdb19316640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:26.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.484+0000 7fdb18b15640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:26.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.484+0000 7fdb18b15640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:26.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.485+0000 7fdb18b15640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:26.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.486+0000 7fdb19316640 10 monclient(hunting): handle_auth_done global_id 22129 payload 274 2024-05-15T13:20:26.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.486+0000 7fdb19316640 1 -- 172.21.15.117:0/1730012654 >> v2:172.21.15.117:3300/0 conn(0x7fdb1406b010 msgr2=0x7fdb14143a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T13:20:26.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.486+0000 7fdb19316640 1 --2- 172.21.15.117:0/1730012654 >> v2:172.21.15.117:3300/0 conn(0x7fdb1406b010 0x7fdb14143a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:26.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.486+0000 7fdb19316640 1 -- 172.21.15.117:0/1730012654 >> v1:172.21.15.117:6789/0 conn(0x7fdb140b30c0 legacy=0x7fdb14071320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:26.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.486+0000 7fdb19316640 10 monclient: _finish_hunting 0 2024-05-15T13:20:26.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.486+0000 7fdb19316640 1 monclient: found mon.noname-c 2024-05-15T13:20:26.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.486+0000 7fdb19316640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:20:26.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.486+0000 7fdb19316640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.178:3300/0 2024-05-15T13:20:26.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.486+0000 7fdb19316640 1 -- 172.21.15.117:0/1730012654 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb140b35a0 con 0x7fdb14143f80 2024-05-15T13:20:26.487 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.488+0000 7fdb19316640 1 --2- 172.21.15.117:0/1730012654 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdb14143f80 0x7fdb14071a30 secure :-1 s=READY pgs=5438 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c009d90 tx=0x7fdb0c02bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=86fa78f93050b8f9 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:26.490 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.491+0000 7fdb137fe640 1 -- 172.21.15.117:0/1730012654 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb0c039070 con 0x7fdb14143f80 2024-05-15T13:20:26.490 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.491+0000 7fdb137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:26.490 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.492+0000 7fdb137fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T13:20:26.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.492+0000 7fdb137fe640 10 monclient: dump: 2024-05-15T13:20:26.491 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:26.491 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:26.491 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:26.491 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:26.491 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:26.491 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:26.491 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:26.491 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:26.491 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:26.491 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:26.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.492+0000 7fdb137fe640 10 monclient: _finish_auth 0 2024-05-15T13:20:26.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.492+0000 7fdb137fe640 10 monclient: _check_auth_tickets 2024-05-15T13:20:26.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.492+0000 7fdb137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:20:26.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.492+0000 7fdb137fe640 1 -- 172.21.15.117:0/1730012654 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb0c02cd00 con 0x7fdb14143f80 2024-05-15T13:20:26.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.492+0000 7fdb137fe640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:20:26.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.492+0000 7fdb137fe640 1 -- 172.21.15.117:0/1730012654 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb0c034660 con 0x7fdb14143f80 2024-05-15T13:20:26.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.492+0000 7fdb137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:26.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.492+0000 7fdb137fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T13:20:26.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.492+0000 7fdb137fe640 10 monclient: dump: 2024-05-15T13:20:26.492 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:26.492 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:26.492 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:26.492 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:26.492 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:26.492 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:26.492 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:26.492 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:26.492 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:26.492 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:26.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.492+0000 7fdb1ad79640 5 monclient: authenticate success, global_id 22129 2024-05-15T13:20:26.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.492+0000 7fdb1ad79640 10 monclient: get_monmap_and_config success 2024-05-15T13:20:26.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.492+0000 7fdb1ad79640 4 set_mon_vals no callback set 2024-05-15T13:20:26.496 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.492+0000 7fdb1ad79640 10 monclient: shutdown 2024-05-15T13:20:26.496 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.492+0000 7fdb1ad79640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:20:26.496 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.492+0000 7fdb1ad79640 1 -- 172.21.15.117:0/1730012654 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdb14143f80 msgr2=0x7fdb14071a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:26.496 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.492+0000 7fdb1ad79640 1 --2- 172.21.15.117:0/1730012654 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdb14143f80 0x7fdb14071a30 secure :-1 s=READY pgs=5438 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c009d90 tx=0x7fdb0c02bf50 comp rx=0 tx=0).stop 2024-05-15T13:20:26.496 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.492+0000 7fdb1ad79640 1 -- 172.21.15.117:0/1730012654 shutdown_connections 2024-05-15T13:20:26.496 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.492+0000 7fdb1ad79640 1 --2- 172.21.15.117:0/1730012654 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdb14143f80 0x7fdb14071a30 unknown :-1 s=CLOSED pgs=5438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:26.496 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.492+0000 7fdb1ad79640 1 --2- 172.21.15.117:0/1730012654 >> v2:172.21.15.117:3300/0 conn(0x7fdb1406b010 0x7fdb14143a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:26.496 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.492+0000 7fdb1ad79640 1 -- 172.21.15.117:0/1730012654 >> 172.21.15.117:0/1730012654 conn(0x7fdb140b19b0 msgr2=0x7fdb140b1dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:20:26.496 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.492+0000 7fdb1ad79640 1 -- 172.21.15.117:0/1730012654 shutdown_connections 2024-05-15T13:20:26.497 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.492+0000 7fdb1ad79640 1 -- 172.21.15.117:0/1730012654 wait complete. 2024-05-15T13:20:26.497 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.493+0000 7fdb1ad79640 10 monclient: build_initial_monmap 2024-05-15T13:20:26.497 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.493+0000 7fdb1ad79640 10 monclient: monmap: 2024-05-15T13:20:26.497 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T13:20:26.497 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T13:20:26.497 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-05-15T13:20:26.497 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-05-15T13:20:26.497 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T13:20:26.497 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:26.497 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T13:20:26.497 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T13:20:26.497 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T13:20:26.497 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:26.497 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.493+0000 7fdb1ad79640 1 librados: starting msgr at 2024-05-15T13:20:26.497 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.493+0000 7fdb1ad79640 1 librados: starting objecter 2024-05-15T13:20:26.497 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.493+0000 7fdb1ad79640 1 Processor -- start 2024-05-15T13:20:26.498 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.493+0000 7fdb1ad79640 1 -- start start 2024-05-15T13:20:26.498 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.493+0000 7fdb1ad79640 1 librados: setting wanted keys 2024-05-15T13:20:26.498 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.493+0000 7fdb1ad79640 1 librados: calling monclient init 2024-05-15T13:20:26.498 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.493+0000 7fdb1ad79640 10 monclient: init 2024-05-15T13:20:26.498 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.494+0000 7fdb1ad79640 10 monclient: _reopen_session rank -1 2024-05-15T13:20:26.498 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.494+0000 7fdb1ad79640 10 monclient: _add_conns ranks=[2,0,1] 2024-05-15T13:20:26.498 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.494+0000 7fdb1ad79640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdb1406b010 0x7fdb1413e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:26.498 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.494+0000 7fdb1ad79640 10 monclient(hunting): picked mon.noname-c con 0x7fdb1406b010 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T13:20:26.498 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.494+0000 7fdb1ad79640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdb140b30c0 0x7fdb140adc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:26.498 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.494+0000 7fdb1ad79640 10 monclient(hunting): picked mon.noname-a con 0x7fdb140b30c0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T13:20:26.498 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.494+0000 7fdb1ad79640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdb14143f80 0x7fdb141d5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:26.498 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.494+0000 7fdb1ad79640 10 monclient(hunting): picked mon.noname-b con 0x7fdb14143f80 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T13:20:26.498 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.494+0000 7fdb1ad79640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:26.498 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.494+0000 7fdb1ad79640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb140b3c20 con 0x7fdb140b30c0 2024-05-15T13:20:26.498 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.494+0000 7fdb1ad79640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:26.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.494+0000 7fdb1ad79640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb140b3aa0 con 0x7fdb14143f80 2024-05-15T13:20:26.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.494+0000 7fdb1ad79640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:26.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.494+0000 7fdb1ad79640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb14071f70 con 0x7fdb1406b010 2024-05-15T13:20:26.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.494+0000 7fdb1ad79640 10 monclient(hunting): _renew_subs 2024-05-15T13:20:26.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.494+0000 7fdb1ad79640 10 monclient(hunting): authenticate will time out at 3589.736347s 2024-05-15T13:20:26.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.494+0000 7fdb13fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdb140b30c0 0x7fdb140adc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:26.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.495+0000 7fdb19316640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdb14143f80 0x7fdb141d5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:26.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.495+0000 7fdb18b15640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdb1406b010 0x7fdb1413e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:26.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.495+0000 7fdb13fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdb140b30c0 0x7fdb140adc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:54768/0 (socket says 172.21.15.117:54768) 2024-05-15T13:20:26.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.495+0000 7fdb13fff640 1 -- 172.21.15.117:0/3206353518 learned_addr learned my addr 172.21.15.117:0/3206353518 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T13:20:26.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.495+0000 7fdb13fff640 10 monclient(hunting): get_auth_request con 0x7fdb140b30c0 auth_method 0 2024-05-15T13:20:26.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.495+0000 7fdb13fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:26.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.495+0000 7fdb13fff640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:26.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.495+0000 7fdb13fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:26.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.495+0000 7fdb19316640 10 monclient(hunting): get_auth_request con 0x7fdb14143f80 auth_method 0 2024-05-15T13:20:26.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.495+0000 7fdb19316640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:26.500 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.495+0000 7fdb19316640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:26.500 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.495+0000 7fdb19316640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:26.500 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.495+0000 7fdb18b15640 10 monclient(hunting): get_auth_request con 0x7fdb1406b010 auth_method 0 2024-05-15T13:20:26.500 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.495+0000 7fdb18b15640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:26.500 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.495+0000 7fdb18b15640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:26.500 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.495+0000 7fdb18b15640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:26.500 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.495+0000 7fdb13fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:26.500 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.495+0000 7fdb13fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:26.500 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.495+0000 7fdb13fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:26.500 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.495+0000 7fdb19316640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:26.500 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.495+0000 7fdb19316640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:26.500 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.495+0000 7fdb19316640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:26.500 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.496+0000 7fdb18b15640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:26.500 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.496+0000 7fdb18b15640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:26.500 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.496+0000 7fdb18b15640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:26.501 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.498+0000 7fdb19316640 10 monclient(hunting): handle_auth_done global_id 22132 payload 824 2024-05-15T13:20:26.501 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.498+0000 7fdb19316640 1 -- 172.21.15.117:0/3206353518 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdb1406b010 msgr2=0x7fdb1413e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:26.501 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.498+0000 7fdb19316640 1 --2- 172.21.15.117:0/3206353518 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdb1406b010 0x7fdb1413e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:26.501 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.498+0000 7fdb19316640 1 -- 172.21.15.117:0/3206353518 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdb140b30c0 msgr2=0x7fdb140adc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:26.502 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.498+0000 7fdb19316640 1 --2- 172.21.15.117:0/3206353518 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdb140b30c0 0x7fdb140adc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:26.502 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.498+0000 7fdb19316640 10 monclient: _finish_hunting 0 2024-05-15T13:20:26.502 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.498+0000 7fdb19316640 1 monclient: found mon.noname-b 2024-05-15T13:20:26.502 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.498+0000 7fdb19316640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:20:26.502 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.498+0000 7fdb19316640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2024-05-15T13:20:26.502 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.498+0000 7fdb19316640 1 -- 172.21.15.117:0/3206353518 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb140ae1a0 con 0x7fdb14143f80 2024-05-15T13:20:26.502 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.500+0000 7fdb19316640 1 --2- 172.21.15.117:0/3206353518 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdb14143f80 0x7fdb141d5a40 secure :-1 s=READY pgs=5439 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c033000 tx=0x7fdb0c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:26.502 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.500+0000 7fdb117fa640 10 client.?.objecter ms_handle_connect 0x7fdb14143f80 2024-05-15T13:20:26.502 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.500+0000 7fdb117fa640 10 client.?.objecter resend_mon_ops 2024-05-15T13:20:26.503 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.504+0000 7fdb117fa640 1 -- 172.21.15.117:0/3206353518 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb0c039050 con 0x7fdb14143f80 2024-05-15T13:20:26.504 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.504+0000 7fdb117fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:26.504 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.504+0000 7fdb117fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T13:20:26.504 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.504+0000 7fdb117fa640 10 monclient: dump: 2024-05-15T13:20:26.504 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:26.504 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:26.504 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:26.504 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:26.504 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:26.504 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:26.504 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:26.504 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:26.504 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:26.504 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:26.504 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.504+0000 7fdb117fa640 10 monclient: _finish_auth 0 2024-05-15T13:20:26.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.504+0000 7fdb117fa640 10 monclient: _check_auth_tickets 2024-05-15T13:20:26.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.504+0000 7fdb117fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:20:26.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.504+0000 7fdb117fa640 1 -- 172.21.15.117:0/3206353518 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb0c045070 con 0x7fdb14143f80 2024-05-15T13:20:26.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.504+0000 7fdb117fa640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:20:26.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.504+0000 7fdb117fa640 1 -- 172.21.15.117:0/3206353518 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb0c009450 con 0x7fdb14143f80 2024-05-15T13:20:26.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.504+0000 7fdb117fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:26.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.504+0000 7fdb117fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T13:20:26.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.504+0000 7fdb117fa640 10 monclient: dump: 2024-05-15T13:20:26.505 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:26.505 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:26.505 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:26.505 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:26.505 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:26.505 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:26.505 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:26.506 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:26.506 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:26.506 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:26.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.504+0000 7fdb1ad79640 5 monclient: authenticate success, global_id 22132 2024-05-15T13:20:26.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.504+0000 7fdb1ad79640 10 monclient: _renew_subs 2024-05-15T13:20:26.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.504+0000 7fdb1ad79640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T13:20:26.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.504+0000 7fdb1ad79640 1 -- 172.21.15.117:0/3206353518 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb140ae650 con 0x7fdb14143f80 2024-05-15T13:20:26.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.504+0000 7fdb12ffd640 4 set_mon_vals no callback set 2024-05-15T13:20:26.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.504+0000 7fdb1ad79640 10 client.22132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:20:26.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.504+0000 7fdb1ad79640 10 monclient: _renew_subs 2024-05-15T13:20:26.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.504+0000 7fdb1ad79640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T13:20:26.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.504+0000 7fdb1ad79640 1 -- 172.21.15.117:0/3206353518 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb14143160 con 0x7fdb14143f80 2024-05-15T13:20:26.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.504+0000 7fdb1ad79640 1 librados: init done 2024-05-15T13:20:26.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.505+0000 7fdb1ad79640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T13:20:26.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.505+0000 7fdb1ad79640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:20:26.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.505+0000 7fdb1ad79640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T13:20:26.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.505+0000 7fdb1ad79640 1 -- 172.21.15.117:0/3206353518 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb140ae880 con 0x7fdb14143f80 2024-05-15T13:20:26.522 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.524+0000 7fdb117fa640 1 -- 172.21.15.117:0/3206353518 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fdb0c03e850 con 0x7fdb14143f80 2024-05-15T13:20:26.523 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.524+0000 7fdb117fa640 1 --2- 172.21.15.117:0/3206353518 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fdaf806ceb0 0x7fdaf806f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:26.523 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.524+0000 7fdb18b15640 1 --2- 172.21.15.117:0/3206353518 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fdaf806ceb0 0x7fdaf806f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:26.523 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.524+0000 7fdb18b15640 10 monclient: get_auth_request con 0x7fdaf806ceb0 auth_method 0 2024-05-15T13:20:26.523 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.525+0000 7fdb117fa640 10 client.22132.objecter ms_handle_connect 0x7fdaf806ceb0 2024-05-15T13:20:26.523 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.525+0000 7fdb18b15640 1 --2- 172.21.15.117:0/3206353518 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fdaf806ceb0 0x7fdaf806f370 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fdb08009b40 tx=0x7fdb08009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:26.525 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.527+0000 7fdb117fa640 1 -- 172.21.15.117:0/3206353518 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(1224..1224 src has 912..1224) v4 ==== 6365+0+0 (secure 0 0 0) 0x7fdb0c07a2c0 con 0x7fdb14143f80 2024-05-15T13:20:26.525 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.527+0000 7fdb117fa640 10 client.22132.objecter ms_dispatch 0x7fdb14000b90 osd_map(1224..1224 src has 912..1224) v4 2024-05-15T13:20:26.525 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.527+0000 7fdb117fa640 3 client.22132.objecter handle_osd_map got epochs [1224,1224] > 0 2024-05-15T13:20:26.525 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.527+0000 7fdb117fa640 3 client.22132.objecter handle_osd_map decoding full epoch 1224 2024-05-15T13:20:26.526 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.527+0000 7fdb117fa640 20 client.22132.objecter dump_active .. 0 homeless 2024-05-15T13:20:26.764 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.765+0000 7fdb117fa640 1 -- 172.21.15.117:0/3206353518 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fdb0c07c220 con 0x7fdb14143f80 2024-05-15T13:20:26.764 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.765+0000 7fdb117fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:20:26.764 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.765+0000 7fdb117fa640 10 monclient: _finish_command 1 = system:0 2024-05-15T13:20:26.976 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.977+0000 7fdb1ad79640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-05-15T13:20:26.976 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.977+0000 7fdb1ad79640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-15T13:20:26.976 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.977+0000 7fdb1ad79640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T13:20:26.976 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:26.977+0000 7fdb1ad79640 1 -- 172.21.15.117:0/3206353518 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdb1404f0e0 con 0x7fdb14143f80 2024-05-15T13:20:26.988 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:38:34.158 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:20:26.993 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:38:34.163 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:20:27.013 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:38:34.183 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:20:27.035 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:38:34.198 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:20:27.046 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:38:34.217 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:20:27.048 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:38:34.218 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:20:27.068 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:38:34.238 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:20:27.080 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:27.082+0000 7fdb117fa640 1 -- 172.21.15.117:0/3206353518 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v1225) v1 ==== 74+0+17822 (secure 0 0 0) 0x7fdb0c0802c0 con 0x7fdb14143f80 2024-05-15T13:20:27.080 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:27.082+0000 7fdb117fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-15T13:20:27.081 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:27.082+0000 7fdb117fa640 10 monclient: _finish_command 2 = system:0 2024-05-15T13:20:27.081 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T13:20:27.081 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":1225,"fsid":"8817f6cf-d933-4825-b972-575ef59f5ba9","created":"2024-05-15T12:42:04.264009+0000","modified":"2024-05-15T13:20:26.822989+0000","last_up_change":"2024-05-15T12:43:01.039161+0000","last_in_change":"2024-05-15T12:42:09.879357+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":105,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":341,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-15T12:42:52.250376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-05-15T12:43:06.641055+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"ceph_test_rados_api_asio","create_time":"2024-05-15T12:44:45.901956+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aaffd6d9-1a00-4793-9d1f-65a30501bec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":1220,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":1673738625},{"type":"v1","addr":"172.21.15.117:6827","nonce":1673738625}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":1673738625},{"type":"v1","addr":"172.21.15.117:6829","nonce":1673738625}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":1673738625},{"type":"v1","addr":"172.21.15.117:6833","nonce":1673738625}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":1673738625},{"type":"v1","addr":"172.21.15.117:6831","nonce":1673738625}]},"public_addr":"172.21.15.117:6827/1673738625","cluster_addr":"172.21.15.117:6829/1673738625","heartbeat_back_addr":"172.21.15.117:6833/1673738625","heartbeat_front_addr":"172.21.15.117:6831/1673738625","state":["exists","up"]},{"osd":1,"uuid":"efd4e8ed-699c-4ccb-b989-cfadc7aaed35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":1206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3864975175},{"type":"v1","addr":"172.21.15.117:6819","nonce":3864975175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3864975175},{"type":"v1","addr":"172.21.15.117:6821","nonce":3864975175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3864975175},{"type":"v1","addr":"172.21.15.117:6825","nonce":3864975175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3864975175},{"type":"v1","addr":"172.21.15.117:6823","nonce":3864975175}]},"public_addr":"172.21.15.117:6819/3864975175","cluster_addr":"172.21.15.117:6821/3864975175","heartbeat_back_addr":"172.21.15.117:6825/3864975175","heartbeat_front_addr":"172.21.15.117:6823/3864975175","state":["exists","up"]},{"osd":2,"uuid":"708e4e3e-7e79-480f-9dbf-7b972c2f2811","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":1213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1425835241},{"type":"v1","addr":"172.21.15.117:6805","nonce":1425835241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1425835241},{"type":"v1","addr":"172.21.15.117:6811","nonce":1425835241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":1425835241},{"type":"v1","addr":"172.21.15.117:6817","nonce":1425835241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6813","nonce":1425835241},{"type":"v1","addr":"172.21.15.117:6815","nonce":1425835241}]},"public_addr":"172.21.15.117:6805/1425835241","cluster_addr":"172.21.15.117:6811/1425835241","heartbeat_back_addr":"172.21.15.117:6817/1425835241","heartbeat_front_addr":"172.21.15.117:6815/1425835241","state":["exists","up"]},{"osd":3,"uuid":"e7b5d7d7-515c-42d9-9c41-b1ca3977865e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":1220,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3842021926},{"type":"v1","addr":"172.21.15.117:6803","nonce":3842021926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3842021926},{"type":"v1","addr":"172.21.15.117:6807","nonce":3842021926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":3842021926},{"type":"v1","addr":"172.21.15.117:6814","nonce":3842021926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":3842021926},{"type":"v1","addr":"172.21.15.117:6809","nonce":3842021926}]},"public_addr":"172.21.15.117:6803/3842021926","cluster_addr":"172.21.15.117:6807/3842021926","heartbeat_back_addr":"172.21.15.117:6814/3842021926","heartbeat_front_addr":"172.21.15.117:6809/3842021926","state":["exists","up"]},{"osd":4,"uuid":"fa647c2c-942b-4be3-afb9-6dba6b0a8cf3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":1220,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":248472851},{"type":"v1","addr":"172.21.15.178:6825","nonce":248472851}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":248472851},{"type":"v1","addr":"172.21.15.178:6827","nonce":248472851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":248472851},{"type":"v1","addr":"172.21.15.178:6831","nonce":248472851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":248472851},{"type":"v1","addr":"172.21.15.178:6829","nonce":248472851}]},"public_addr":"172.21.15.178:6825/248472851","cluster_addr":"172.21.15.178:6827/248472851","heartbeat_back_addr":"172.21.15.178:6831/248472851","heartbeat_front_addr":"172.21.15.178:6829/248472851","state":["exists","up"]},{"osd":5,"uuid":"c5f9bc37-4bd6-4643-8f0d-e53c5f2c8782","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":1220,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":3713764498},{"type":"v1","addr":"172.21.15.178:6817","nonce":3713764498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":3713764498},{"type":"v1","addr":"172.21.15.178:6819","nonce":3713764498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":3713764498},{"type":"v1","addr":"172.21.15.178:6823","nonce":3713764498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":3713764498},{"type":"v1","addr":"172.21.15.178:6821","nonce":3713764498}]},"public_addr":"172.21.15.178:6817/3713764498","cluster_addr":"172.21.15.178:6819/3713764498","heartbeat_back_addr":"172.21.15.178:6823/3713764498","heartbeat_front_addr":"172.21.15.178:6821/3713764498","state":["exists","up"]},{"osd":6,"uuid":"d5af075d-3eaa-4cb5-b6e9-6f01c201f2ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":1220,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1383036222},{"type":"v1","addr":"172.21.15.178:6801","nonce":1383036222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1383036222},{"type":"v1","addr":"172.21.15.178:6803","nonce":1383036222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1383036222},{"type":"v1","addr":"172.21.15.178:6807","nonce":1383036222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1383036222},{"type":"v1","addr":"172.21.15.178:6805","nonce":1383036222}]},"public_addr":"172.21.15.178:6801/1383036222","cluster_addr":"172.21.15.178:6803/1383036222","heartbeat_back_addr":"172.21.15.178:6807/1383036222","heartbeat_front_addr":"172.21.15.178:6805/1383036222","state":["exists","up"]},{"osd":7,"uuid":"323233c7-a3af-4fb0-ac70-de523a279b44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":1220,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":3049776526},{"type":"v1","addr":"172.21.15.178:6809","nonce":3049776526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":3049776526},{"type":"v1","addr":"172.21.15.178:6811","nonce":3049776526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":3049776526},{"type":"v1","addr":"172.21.15.178:6815","nonce":3049776526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":3049776526},{"type":"v1","addr":"172.21.15.178:6813","nonce":3049776526}]},"public_addr":"172.21.15.178:6809/3049776526","cluster_addr":"172.21.15.178:6811/3049776526","heartbeat_back_addr":"172.21.15.178:6815/3049776526","heartbeat_front_addr":"172.21.15.178:6813/3049776526","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:42:48.855951+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:42:49.166018+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:42:48.891180+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:42:48.243508+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:42:47.750264+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:42:47.225543+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:42:47.826018+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:42:47.675658+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":3,"length":4}]},{"pool":11,"snaps":[{"begin":4,"length":1}]},{"pool":25,"snaps":[{"begin":1,"length":1}]},{"pool":71,"snaps":[{"begin":1,"length":1}]},{"pool":97,"snaps":[{"begin":1,"length":1}]},{"pool":99,"snaps":[{"begin":4,"length":1},{"begin":7,"length":2}]},{"pool":106,"snaps":[{"begin":1,"length":2}]},{"pool":119,"snaps":[{"begin":3,"length":1}]},{"pool":121,"snaps":[{"begin":2,"length":2}]},{"pool":125,"snaps":[{"begin":1,"length":4}]},{"pool":127,"snaps":[{"begin":2,"length":3}]},{"pool":134,"snaps":[{"begin":2,"length":1}]},{"pool":136,"snaps":[{"begin":2,"length":2}]},{"pool":139,"snaps":[{"begin":2,"length":3}]},{"pool":140,"snaps":[{"begin":2,"length":4}]},{"pool":145,"snaps":[{"begin":4,"length":1},{"begin":6,"length":2}]},{"pool":146,"snaps":[{"begin":3,"length":4}]},{"pool":155,"snaps":[{"begin":7,"length":1}]},{"pool":159,"snaps":[{"begin":8,"length":1}]},{"pool":161,"snaps":[{"begin":10,"length":1}]},{"pool":162,"snaps":[{"begin":11,"length":2}]},{"pool":165,"snaps":[{"begin":14,"length":1}]},{"pool":208,"snaps":[{"begin":2,"length":1}]},{"pool":209,"snaps":[{"begin":3,"length":1}]},{"pool":212,"snaps":[{"begin":4,"length":1}]},{"pool":218,"snaps":[{"begin":6,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-15T13:20:27.082 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:38:34.245 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:20:27.083 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:27.084+0000 7fdb1ad79640 10 librados: watch_flush enter 2024-05-15T13:20:27.083 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:27.084+0000 7fdb1ad79640 10 librados: watch_flush exit 2024-05-15T13:20:27.084 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:27.084+0000 7fdb1ad79640 20 client.22132.objecter shutdown clearing up homeless session... 2024-05-15T13:20:27.084 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:27.084+0000 7fdb1ad79640 10 client.22132.objecter successfully canceled tick 2024-05-15T13:20:27.084 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:27.084+0000 7fdb1ad79640 1 -- 172.21.15.117:0/3206353518 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fdaf806ceb0 msgr2=0x7fdaf806f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:27.084 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:27.084+0000 7fdb1ad79640 1 --2- 172.21.15.117:0/3206353518 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fdaf806ceb0 0x7fdaf806f370 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fdb08009b40 tx=0x7fdb08009190 comp rx=0 tx=0).stop 2024-05-15T13:20:27.084 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:27.084+0000 7fdb1ad79640 10 monclient: shutdown 2024-05-15T13:20:27.084 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:27.084+0000 7fdb1ad79640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:20:27.084 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:27.084+0000 7fdb1ad79640 1 -- 172.21.15.117:0/3206353518 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdb14143f80 msgr2=0x7fdb141d5a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:27.084 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:27.084+0000 7fdb1ad79640 1 --2- 172.21.15.117:0/3206353518 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdb14143f80 0x7fdb141d5a40 secure :-1 s=READY pgs=5439 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c033000 tx=0x7fdb0c004a40 comp rx=0 tx=0).stop 2024-05-15T13:20:27.084 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:27.084+0000 7fdb1ad79640 1 -- 172.21.15.117:0/3206353518 shutdown_connections 2024-05-15T13:20:27.084 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:27.084+0000 7fdb1ad79640 1 --2- 172.21.15.117:0/3206353518 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fdaf806ceb0 0x7fdaf806f370 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:27.084 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:27.084+0000 7fdb1ad79640 1 --2- 172.21.15.117:0/3206353518 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdb14143f80 0x7fdb141d5a40 unknown :-1 s=CLOSED pgs=5439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:27.084 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:27.084+0000 7fdb1ad79640 1 --2- 172.21.15.117:0/3206353518 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdb140b30c0 0x7fdb140adc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:27.084 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:27.084+0000 7fdb1ad79640 1 --2- 172.21.15.117:0/3206353518 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdb1406b010 0x7fdb1413e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:27.084 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:27.084+0000 7fdb1ad79640 1 -- 172.21.15.117:0/3206353518 >> 172.21.15.117:0/3206353518 conn(0x7fdb140b19b0 msgr2=0x7fdb140b1d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:20:27.084 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:27.084+0000 7fdb1ad79640 1 -- 172.21.15.117:0/3206353518 shutdown_connections 2024-05-15T13:20:27.085 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:27.084+0000 7fdb1ad79640 1 -- 172.21.15.117:0/3206353518 wait complete. 2024-05-15T13:20:27.085 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:27.084+0000 7fdb1ad79640 1 librados: shutdown 2024-05-15T13:20:27.102 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:38:34.266 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:20:28.097 INFO:tasks.ceph:Scrubbing osd.0 2024-05-15T13:20:28.097 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-05-15T13:20:28.202 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.203+0000 7fe31f332640 10 monclient: get_monmap_and_config 2024-05-15T13:20:28.202 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.203+0000 7fe31f332640 10 monclient: build_initial_monmap 2024-05-15T13:20:28.202 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.203+0000 7fe31f332640 10 monclient: monmap: 2024-05-15T13:20:28.202 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T13:20:28.202 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:28.202 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T13:20:28.204560+0000 2024-05-15T13:20:28.202 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T13:20:28.204560+0000 2024-05-15T13:20:28.202 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T13:20:28.202 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:28.203 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T13:20:28.203 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T13:20:28.203 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T13:20:28.203 INFO:teuthology.orchestra.run.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T13:20:28.203 INFO:teuthology.orchestra.run.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T13:20:28.203 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:28.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.204+0000 7fe31f332640 1 Processor -- start 2024-05-15T13:20:28.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.204+0000 7fe31f332640 1 -- start start 2024-05-15T13:20:28.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.204+0000 7fe31f332640 10 monclient: init 2024-05-15T13:20:28.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.204+0000 7fe31f332640 10 monclient: _reopen_session rank -1 2024-05-15T13:20:28.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.204+0000 7fe31f332640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-05-15T13:20:28.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.204+0000 7fe31f332640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe31806b010 addr v1:172.21.15.117:6789/0 2024-05-15T13:20:28.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.204+0000 7fe31f332640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe3180b30c0 0x7fe31813d040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:28.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.204+0000 7fe31f332640 10 monclient(hunting): picked mon.noname-b con 0x7fe3180b30c0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T13:20:28.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.204+0000 7fe31f332640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe318144150 addr v1:172.21.15.178:6789/0 2024-05-15T13:20:28.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.204+0000 7fe31f332640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3180b4740 con 0x7fe31806b010 2024-05-15T13:20:28.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.204+0000 7fe31f332640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe318072f00 con 0x7fe318144150 2024-05-15T13:20:28.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.204+0000 7fe31f332640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:28.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.204+0000 7fe31f332640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe31806b3f0 con 0x7fe3180b30c0 2024-05-15T13:20:28.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.204+0000 7fe31f332640 10 monclient(hunting): _renew_subs 2024-05-15T13:20:28.206 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.204+0000 7fe31f332640 10 monclient(hunting): authenticate will time out at 3591.446773s 2024-05-15T13:20:28.206 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.205+0000 7fe31d0ce640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fe31806b010 0x7fe318143a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:54084/0 (socket says 172.21.15.117:54084) 2024-05-15T13:20:28.206 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.205+0000 7fe31d0ce640 1 -- 172.21.15.117:0/2895339486 learned_addr learned my addr 172.21.15.117:0/2895339486 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-05-15T13:20:28.206 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.205+0000 7fe31c8cd640 1 --2- 172.21.15.117:0/2895339486 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe3180b30c0 0x7fe31813d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:28.206 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.205+0000 7fe31c8cd640 10 monclient(hunting): get_auth_request con 0x7fe3180b30c0 auth_method 0 2024-05-15T13:20:28.206 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.205+0000 7fe31c8cd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:28.206 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.205+0000 7fe31c8cd640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:28.206 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.205+0000 7fe31c8cd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:28.206 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.206+0000 7fe31c8cd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:28.206 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.206+0000 7fe31c8cd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:28.206 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.206+0000 7fe31c8cd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:28.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.207+0000 7fe30ffff640 1 -- 172.21.15.117:0/2895339486 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2134132730 0 0) 0x7fe314001110 con 0x7fe31806b010 2024-05-15T13:20:28.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.207+0000 7fe30ffff640 10 monclient(hunting): _negotiate 2024-05-15T13:20:28.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.207+0000 7fe30ffff640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:28.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.207+0000 7fe30ffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:28.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.207+0000 7fe30ffff640 10 monclient(hunting): my global_id is 16005 2024-05-15T13:20:28.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.208+0000 7fe30ffff640 1 -- 172.21.15.117:0/2895339486 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe304003520 con 0x7fe31806b010 2024-05-15T13:20:28.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.208+0000 7fe30ffff640 1 -- 172.21.15.117:0/2895339486 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 317264486 0 0) 0x7fe318072f00 con 0x7fe318144150 2024-05-15T13:20:28.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.208+0000 7fe30ffff640 10 monclient(hunting): _negotiate 2024-05-15T13:20:28.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.208+0000 7fe30ffff640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:28.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.208+0000 7fe30ffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:28.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.208+0000 7fe30ffff640 10 monclient(hunting): my global_id is 22135 2024-05-15T13:20:28.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.208+0000 7fe30ffff640 1 -- 172.21.15.117:0/2895339486 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe314001110 con 0x7fe318144150 2024-05-15T13:20:28.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.208+0000 7fe31c8cd640 10 monclient(hunting): handle_auth_done global_id 18716 payload 274 2024-05-15T13:20:28.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.208+0000 7fe31c8cd640 1 -- 172.21.15.117:0/2895339486 >> v1:172.21.15.178:6789/0 conn(0x7fe318144150 legacy=0x7fe31813d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:28.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.208+0000 7fe31c8cd640 1 -- 172.21.15.117:0/2895339486 >> v1:172.21.15.117:6789/0 conn(0x7fe31806b010 legacy=0x7fe318143a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:28.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.208+0000 7fe31c8cd640 10 monclient: _finish_hunting 0 2024-05-15T13:20:28.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.208+0000 7fe31c8cd640 1 monclient: found mon.noname-b 2024-05-15T13:20:28.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.208+0000 7fe31c8cd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:20:28.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.208+0000 7fe31c8cd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.117:3301/0 2024-05-15T13:20:28.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.208+0000 7fe31c8cd640 1 -- 172.21.15.117:0/2895339486 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3180b35a0 con 0x7fe3180b30c0 2024-05-15T13:20:28.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.210+0000 7fe31c8cd640 1 --2- 172.21.15.117:0/2895339486 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe3180b30c0 0x7fe31813d040 secure :-1 s=READY pgs=3380 cs=0 l=1 rev1=1 crypto rx=0x7fe3080097d0 tx=0x7fe30802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb976cfa368d841e server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:28.213 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.213+0000 7fe30ffff640 1 -- 172.21.15.117:0/2895339486 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe308035020 con 0x7fe3180b30c0 2024-05-15T13:20:28.213 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.213+0000 7fe30ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:28.213 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.213+0000 7fe30ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T13:20:28.213 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.213+0000 7fe30ffff640 10 monclient: dump: 2024-05-15T13:20:28.214 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:28.214 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:28.214 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:28.214 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:28.214 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:28.214 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:28.214 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:28.214 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:28.214 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:28.214 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:28.214 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.214+0000 7fe30ffff640 10 monclient: _finish_auth 0 2024-05-15T13:20:28.214 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.214+0000 7fe30ffff640 10 monclient: _check_auth_tickets 2024-05-15T13:20:28.214 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.214+0000 7fe30ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:20:28.214 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.214+0000 7fe30ffff640 1 -- 172.21.15.117:0/2895339486 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe308031890 con 0x7fe3180b30c0 2024-05-15T13:20:28.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.214+0000 7fe31f332640 5 monclient: authenticate success, global_id 18716 2024-05-15T13:20:28.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.214+0000 7fe31f332640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T13:20:28.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.214+0000 7fe30ffff640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:20:28.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.214+0000 7fe31f332640 10 monclient: get_monmap_and_config success 2024-05-15T13:20:28.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.214+0000 7fe31f332640 4 set_mon_vals no callback set 2024-05-15T13:20:28.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.214+0000 7fe30ffff640 1 -- 172.21.15.117:0/2895339486 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe308031c60 con 0x7fe3180b30c0 2024-05-15T13:20:28.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.214+0000 7fe30ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:28.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.214+0000 7fe30ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T13:20:28.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.214+0000 7fe30ffff640 10 monclient: dump: 2024-05-15T13:20:28.215 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:28.215 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:28.215 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:28.215 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:28.215 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:28.215 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:28.216 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:28.216 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:28.216 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:28.216 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:28.216 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.214+0000 7fe31f332640 10 monclient: shutdown 2024-05-15T13:20:28.216 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.214+0000 7fe31f332640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:20:28.216 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.214+0000 7fe31f332640 1 -- 172.21.15.117:0/2895339486 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe3180b30c0 msgr2=0x7fe31813d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:28.216 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.214+0000 7fe31f332640 1 --2- 172.21.15.117:0/2895339486 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe3180b30c0 0x7fe31813d040 secure :-1 s=READY pgs=3380 cs=0 l=1 rev1=1 crypto rx=0x7fe3080097d0 tx=0x7fe30802b930 comp rx=0 tx=0).stop 2024-05-15T13:20:28.216 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.214+0000 7fe31f332640 1 -- 172.21.15.117:0/2895339486 shutdown_connections 2024-05-15T13:20:28.216 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.214+0000 7fe31f332640 1 --2- 172.21.15.117:0/2895339486 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe3180b30c0 0x7fe31813d040 unknown :-1 s=CLOSED pgs=3380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:28.216 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.214+0000 7fe31f332640 1 -- 172.21.15.117:0/2895339486 >> 172.21.15.117:0/2895339486 conn(0x7fe3180b19b0 msgr2=0x7fe3180b1dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:20:28.216 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.214+0000 7fe31f332640 1 -- 172.21.15.117:0/2895339486 shutdown_connections 2024-05-15T13:20:28.216 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.214+0000 7fe31f332640 1 -- 172.21.15.117:0/2895339486 wait complete. 2024-05-15T13:20:28.219 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.215+0000 7fe31f332640 10 monclient: build_initial_monmap 2024-05-15T13:20:28.219 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.215+0000 7fe31f332640 10 monclient: monmap: 2024-05-15T13:20:28.219 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T13:20:28.219 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T13:20:28.220 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-05-15T13:20:28.220 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-05-15T13:20:28.220 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T13:20:28.220 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:28.220 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T13:20:28.220 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T13:20:28.220 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T13:20:28.220 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:28.220 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.215+0000 7fe31f332640 1 librados: starting msgr at 2024-05-15T13:20:28.220 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.215+0000 7fe31f332640 1 librados: starting objecter 2024-05-15T13:20:28.220 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.216+0000 7fe31f332640 1 Processor -- start 2024-05-15T13:20:28.220 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.216+0000 7fe31f332640 1 -- start start 2024-05-15T13:20:28.220 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.216+0000 7fe31f332640 1 librados: setting wanted keys 2024-05-15T13:20:28.220 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.216+0000 7fe31f332640 1 librados: calling monclient init 2024-05-15T13:20:28.220 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.216+0000 7fe31f332640 10 monclient: init 2024-05-15T13:20:28.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.216+0000 7fe31f332640 10 monclient: _reopen_session rank -1 2024-05-15T13:20:28.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.216+0000 7fe31f332640 10 monclient: _add_conns ranks=[0,2,1] 2024-05-15T13:20:28.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.216+0000 7fe31f332640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe31806b010 0x7fe3180ae0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:28.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.216+0000 7fe31f332640 10 monclient(hunting): picked mon.noname-a con 0x7fe31806b010 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T13:20:28.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.216+0000 7fe31f332640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe3180b30c0 0x7fe31806fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:28.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.216+0000 7fe31f332640 10 monclient(hunting): picked mon.noname-c con 0x7fe3180b30c0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T13:20:28.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.216+0000 7fe31f332640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe318144150 0x7fe3180700f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:28.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.216+0000 7fe31f332640 10 monclient(hunting): picked mon.noname-b con 0x7fe318144150 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T13:20:28.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.216+0000 7fe31f332640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:28.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.216+0000 7fe31f332640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe31813dc90 con 0x7fe31806b010 2024-05-15T13:20:28.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.216+0000 7fe31f332640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:28.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.216+0000 7fe31f332640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3180b4b30 con 0x7fe318144150 2024-05-15T13:20:28.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.216+0000 7fe31f332640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:28.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.216+0000 7fe31f332640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe31813de10 con 0x7fe3180b30c0 2024-05-15T13:20:28.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.216+0000 7fe31f332640 10 monclient(hunting): _renew_subs 2024-05-15T13:20:28.222 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.216+0000 7fe31f332640 10 monclient(hunting): authenticate will time out at 3591.458489s 2024-05-15T13:20:28.222 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.217+0000 7fe31c8cd640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe3180b30c0 0x7fe31806fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:28.222 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.217+0000 7fe31d0ce640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe31806b010 0x7fe3180ae0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:28.222 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.217+0000 7fe31d8cf640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe318144150 0x7fe3180700f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:28.222 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.217+0000 7fe31d0ce640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe31806b010 0x7fe3180ae0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:54770/0 (socket says 172.21.15.117:54770) 2024-05-15T13:20:28.222 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.217+0000 7fe31d0ce640 1 -- 172.21.15.117:0/1001050348 learned_addr learned my addr 172.21.15.117:0/1001050348 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T13:20:28.222 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.217+0000 7fe31d0ce640 10 monclient(hunting): get_auth_request con 0x7fe31806b010 auth_method 0 2024-05-15T13:20:28.222 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.217+0000 7fe31d0ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:28.222 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.217+0000 7fe31d0ce640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:28.222 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.217+0000 7fe31d0ce640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:28.222 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.217+0000 7fe31d8cf640 10 monclient(hunting): get_auth_request con 0x7fe318144150 auth_method 0 2024-05-15T13:20:28.223 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.217+0000 7fe31d8cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:28.223 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.217+0000 7fe31d8cf640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:28.223 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.217+0000 7fe31d8cf640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:28.223 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.217+0000 7fe31c8cd640 10 monclient(hunting): get_auth_request con 0x7fe3180b30c0 auth_method 0 2024-05-15T13:20:28.223 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.217+0000 7fe31c8cd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:28.223 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.217+0000 7fe31c8cd640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:28.223 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.217+0000 7fe31c8cd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:28.223 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.217+0000 7fe31d0ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:28.223 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.217+0000 7fe31d0ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:28.223 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.217+0000 7fe31d0ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:28.223 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.218+0000 7fe31c8cd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:28.223 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.218+0000 7fe31c8cd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:28.223 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.218+0000 7fe31c8cd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:28.223 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.218+0000 7fe31d8cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:28.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.218+0000 7fe31d8cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:28.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.218+0000 7fe31d8cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:28.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.220+0000 7fe31d0ce640 10 monclient(hunting): handle_auth_done global_id 16008 payload 824 2024-05-15T13:20:28.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.220+0000 7fe31d0ce640 1 -- 172.21.15.117:0/1001050348 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe3180b30c0 msgr2=0x7fe31806fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:28.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.220+0000 7fe31d0ce640 1 --2- 172.21.15.117:0/1001050348 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe3180b30c0 0x7fe31806fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:28.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.220+0000 7fe31d0ce640 1 -- 172.21.15.117:0/1001050348 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe318144150 msgr2=0x7fe3180700f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:28.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.220+0000 7fe31d0ce640 1 --2- 172.21.15.117:0/1001050348 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe318144150 0x7fe3180700f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:28.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.220+0000 7fe31d0ce640 10 monclient: _finish_hunting 0 2024-05-15T13:20:28.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.220+0000 7fe31d0ce640 1 monclient: found mon.noname-a 2024-05-15T13:20:28.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.220+0000 7fe31d0ce640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:20:28.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.220+0000 7fe31d0ce640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.117:3300/0 2024-05-15T13:20:28.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.220+0000 7fe31d0ce640 1 -- 172.21.15.117:0/1001050348 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe31806f630 con 0x7fe31806b010 2024-05-15T13:20:28.226 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.223+0000 7fe31d0ce640 1 --2- 172.21.15.117:0/1001050348 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe31806b010 0x7fe3180ae0b0 secure :-1 s=READY pgs=3293 cs=0 l=1 rev1=1 crypto rx=0x7fe31400c6b0 tx=0x7fe31400cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:28.226 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.223+0000 7fe30dffb640 10 client.?.objecter ms_handle_connect 0x7fe31806b010 2024-05-15T13:20:28.226 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.223+0000 7fe30dffb640 10 client.?.objecter resend_mon_ops 2024-05-15T13:20:28.226 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.226+0000 7fe30dffb640 1 -- 172.21.15.117:0/1001050348 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe314017070 con 0x7fe31806b010 2024-05-15T13:20:28.226 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.226+0000 7fe30dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:28.226 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.226+0000 7fe30dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T13:20:28.226 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.226+0000 7fe30dffb640 10 monclient: dump: 2024-05-15T13:20:28.226 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:28.226 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:28.226 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:28.226 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:28.227 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:28.227 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:28.227 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:28.227 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:28.227 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:28.227 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:28.227 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.227+0000 7fe30dffb640 10 monclient: _finish_auth 0 2024-05-15T13:20:28.227 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.227+0000 7fe30dffb640 10 monclient: _check_auth_tickets 2024-05-15T13:20:28.227 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.227+0000 7fe30dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:20:28.227 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.227+0000 7fe30dffb640 1 -- 172.21.15.117:0/1001050348 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe314005af0 con 0x7fe31806b010 2024-05-15T13:20:28.227 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.227+0000 7fe30dffb640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:20:28.227 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.227+0000 7fe30dffb640 1 -- 172.21.15.117:0/1001050348 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe314007070 con 0x7fe31806b010 2024-05-15T13:20:28.227 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.227+0000 7fe31f332640 5 monclient: authenticate success, global_id 16008 2024-05-15T13:20:28.227 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.227+0000 7fe30dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:28.227 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.227+0000 7fe30dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T13:20:28.228 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.227+0000 7fe30dffb640 10 monclient: dump: 2024-05-15T13:20:28.228 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:28.228 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:28.228 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:28.228 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:28.228 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:28.228 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:28.228 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:28.228 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:28.228 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:28.228 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:28.228 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.227+0000 7fe31f332640 10 monclient: _renew_subs 2024-05-15T13:20:28.228 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.227+0000 7fe31f332640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T13:20:28.228 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.227+0000 7fe31f332640 1 -- 172.21.15.117:0/1001050348 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe31813e320 con 0x7fe31806b010 2024-05-15T13:20:28.229 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.227+0000 7fe30f7fe640 4 set_mon_vals no callback set 2024-05-15T13:20:28.229 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.227+0000 7fe31f332640 10 client.16008.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:20:28.229 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.227+0000 7fe31f332640 10 monclient: _renew_subs 2024-05-15T13:20:28.229 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.227+0000 7fe31f332640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T13:20:28.229 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.227+0000 7fe31f332640 1 -- 172.21.15.117:0/1001050348 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe318070630 con 0x7fe31806b010 2024-05-15T13:20:28.229 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.227+0000 7fe31f332640 1 librados: init done 2024-05-15T13:20:28.229 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.227+0000 7fe31f332640 10 client.16008.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:20:28.229 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.228+0000 7fe31f332640 15 client.16008.objecter _session_command_op_assign -1 1 2024-05-15T13:20:28.230 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.228+0000 7fe31f332640 20 client.16008.objecter _get_session osd=-1 returning homeless 2024-05-15T13:20:28.230 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.228+0000 7fe31f332640 10 client.16008.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:20:28.230 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.228+0000 7fe31f332640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T13:20:28.230 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.228+0000 7fe31f332640 1 -- 172.21.15.117:0/1001050348 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe31804f0e0 con 0x7fe31806b010 2024-05-15T13:20:28.244 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.246+0000 7fe30dffb640 1 -- 172.21.15.117:0/1001050348 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fe31401a070 con 0x7fe31806b010 2024-05-15T13:20:28.245 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.246+0000 7fe30dffb640 1 --2- 172.21.15.117:0/1001050348 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fe2fc06ce60 0x7fe2fc06f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:28.245 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.247+0000 7fe31c8cd640 1 --2- 172.21.15.117:0/1001050348 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fe2fc06ce60 0x7fe2fc06f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:28.245 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.247+0000 7fe31c8cd640 10 monclient: get_auth_request con 0x7fe2fc06ce60 auth_method 0 2024-05-15T13:20:28.246 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.247+0000 7fe30dffb640 10 client.16008.objecter ms_handle_connect 0x7fe2fc06ce60 2024-05-15T13:20:28.246 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.247+0000 7fe31c8cd640 1 --2- 172.21.15.117:0/1001050348 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fe2fc06ce60 0x7fe2fc06f320 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fe308036000 tx=0x7fe308008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:28.248 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.249+0000 7fe30dffb640 1 -- 172.21.15.117:0/1001050348 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(1225..1225 src has 912..1225) v4 ==== 6365+0+0 (secure 0 0 0) 0x7fe314056b50 con 0x7fe31806b010 2024-05-15T13:20:28.248 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.249+0000 7fe30dffb640 10 client.16008.objecter ms_dispatch 0x7fe318000b90 osd_map(1225..1225 src has 912..1225) v4 2024-05-15T13:20:28.248 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.249+0000 7fe30dffb640 3 client.16008.objecter handle_osd_map got epochs [1225,1225] > 0 2024-05-15T13:20:28.248 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.249+0000 7fe30dffb640 3 client.16008.objecter handle_osd_map decoding full epoch 1225 2024-05-15T13:20:28.248 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.249+0000 7fe30dffb640 10 client.16008.objecter checking command 1 2024-05-15T13:20:28.249 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.249+0000 7fe30dffb640 1 --2- 172.21.15.117:0/1001050348 >> [v2:172.21.15.117:6826/1673738625,v1:172.21.15.117:6827/1673738625] conn(0x7fe2fc074b70 0x7fe2fc077010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:28.249 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.249+0000 7fe30dffb640 20 client.16008.objecter _get_session s=0x7fe2fc0744a0 osd=0 3 2024-05-15T13:20:28.249 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.249+0000 7fe30dffb640 20 client.16008.objecter put_session s=0x7fe2fc0744a0 osd=0 3 2024-05-15T13:20:28.249 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.249+0000 7fe30dffb640 15 client.16008.objecter _session_command_op_remove -1 1 2024-05-15T13:20:28.249 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.249+0000 7fe30dffb640 20 client.16008.objecter _get_session s=0x7fe2fc0744a0 osd=0 3 2024-05-15T13:20:28.249 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.249+0000 7fe30dffb640 20 client.16008.objecter get_session s=0x7fe2fc0744a0 osd=0 3 2024-05-15T13:20:28.249 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.249+0000 7fe30dffb640 15 client.16008.objecter _session_command_op_assign 0 1 2024-05-15T13:20:28.249 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.249+0000 7fe30dffb640 20 client.16008.objecter put_session s=0x7fe2fc0744a0 osd=0 4 2024-05-15T13:20:28.249 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.249+0000 7fe30dffb640 10 client.16008.objecter _send_command 1 2024-05-15T13:20:28.249 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.249+0000 7fe30dffb640 1 -- 172.21.15.117:0/1001050348 --> [v2:172.21.15.117:6826/1673738625,v1:172.21.15.117:6827/1673738625] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe314015e70 con 0x7fe2fc074b70 2024-05-15T13:20:28.249 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.249+0000 7fe30dffb640 20 client.16008.objecter dump_active .. 0 homeless 2024-05-15T13:20:28.250 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.250+0000 7fe31d8cf640 1 --2- 172.21.15.117:0/1001050348 >> [v2:172.21.15.117:6826/1673738625,v1:172.21.15.117:6827/1673738625] conn(0x7fe2fc074b70 0x7fe2fc077010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:28.250 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.250+0000 7fe31d8cf640 10 monclient: get_auth_request con 0x7fe2fc074b70 auth_method 0 2024-05-15T13:20:28.250 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.251+0000 7fe30dffb640 1 -- 172.21.15.117:0/1001050348 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_get_version_reply(handle=1 version=1225) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe314054d50 con 0x7fe31806b010 2024-05-15T13:20:28.250 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.251+0000 7fe30dffb640 10 monclient: handle_get_version_reply finishing 1 version 1225 2024-05-15T13:20:28.252 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.253+0000 7fe31d8cf640 1 --2- 172.21.15.117:0/1001050348 >> [v2:172.21.15.117:6826/1673738625,v1:172.21.15.117:6827/1673738625] conn(0x7fe2fc074b70 0x7fe2fc077010 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:28.252 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.253+0000 7fe30dffb640 10 client.16008.objecter ms_handle_connect 0x7fe2fc074b70 2024-05-15T13:20:28.276 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.277+0000 7fe30dffb640 1 -- 172.21.15.117:0/1001050348 <== osd.0 v2:172.21.15.117:6826/1673738625 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe314015e70 con 0x7fe2fc074b70 2024-05-15T13:20:28.276 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.277+0000 7fe30dffb640 10 client.16008.objecter ms_dispatch 0x7fe318000b90 command_reply(tid 1: 0 ) v1 2024-05-15T13:20:28.276 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.277+0000 7fe30dffb640 10 client.16008.objecter _finish_command 1 = system:0 2024-05-15T13:20:28.276 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.277+0000 7fe30dffb640 20 client.16008.objecter put_session s=0x7fe2fc0744a0 osd=0 4 2024-05-15T13:20:28.276 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.277+0000 7fe30dffb640 15 client.16008.objecter _session_command_op_remove 0 1 2024-05-15T13:20:28.289 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.290+0000 7fe31f332640 10 client.16008.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-05-15T13:20:28.289 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.290+0000 7fe31f332640 15 client.16008.objecter _session_command_op_assign -1 2 2024-05-15T13:20:28.289 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.290+0000 7fe31f332640 20 client.16008.objecter _get_session s=0x7fe2fc0744a0 osd=0 3 2024-05-15T13:20:28.289 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.290+0000 7fe31f332640 20 client.16008.objecter put_session s=0x7fe2fc0744a0 osd=0 3 2024-05-15T13:20:28.289 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.290+0000 7fe31f332640 20 client.16008.objecter _get_session s=0x7fe2fc0744a0 osd=0 3 2024-05-15T13:20:28.289 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.290+0000 7fe31f332640 15 client.16008.objecter _session_command_op_remove -1 2 2024-05-15T13:20:28.289 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.290+0000 7fe31f332640 20 client.16008.objecter get_session s=0x7fe2fc0744a0 osd=0 3 2024-05-15T13:20:28.289 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.290+0000 7fe31f332640 15 client.16008.objecter _session_command_op_assign 0 2 2024-05-15T13:20:28.289 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.290+0000 7fe31f332640 20 client.16008.objecter put_session s=0x7fe2fc0744a0 osd=0 4 2024-05-15T13:20:28.289 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.290+0000 7fe31f332640 10 client.16008.objecter _send_command 2 2024-05-15T13:20:28.289 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.290+0000 7fe31f332640 1 -- 172.21.15.117:0/1001050348 --> [v2:172.21.15.117:6826/1673738625,v1:172.21.15.117:6827/1673738625] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fe31813c420 con 0x7fe2fc074b70 2024-05-15T13:20:28.308 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.309+0000 7fe30dffb640 1 -- 172.21.15.117:0/1001050348 <== osd.0 v2:172.21.15.117:6826/1673738625 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fe31813c420 con 0x7fe2fc074b70 2024-05-15T13:20:28.308 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.309+0000 7fe30dffb640 10 client.16008.objecter ms_dispatch 0x7fe318000b90 command_reply(tid 2: 0 ) v1 2024-05-15T13:20:28.308 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.309+0000 7fe30dffb640 10 client.16008.objecter _finish_command 2 = system:0 2024-05-15T13:20:28.308 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.309+0000 7fe30dffb640 20 client.16008.objecter put_session s=0x7fe2fc0744a0 osd=0 4 2024-05-15T13:20:28.308 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.309+0000 7fe30dffb640 15 client.16008.objecter _session_command_op_remove 0 2 2024-05-15T13:20:28.309 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-05-15T13:20:28.309 INFO:teuthology.orchestra.run.smithi117.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-15T13:20:28.309 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-05-15T13:20:28.310 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.310+0000 7fe31f332640 10 librados: watch_flush enter 2024-05-15T13:20:28.310 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.310+0000 7fe31f332640 10 librados: watch_flush exit 2024-05-15T13:20:28.310 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.310+0000 7fe31f332640 10 client.16008.objecter close_session for osd.0 2024-05-15T13:20:28.310 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.310+0000 7fe31f332640 1 -- 172.21.15.117:0/1001050348 >> [v2:172.21.15.117:6826/1673738625,v1:172.21.15.117:6827/1673738625] conn(0x7fe2fc074b70 msgr2=0x7fe2fc077010 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:28.310 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.310+0000 7fe31f332640 1 --2- 172.21.15.117:0/1001050348 >> [v2:172.21.15.117:6826/1673738625,v1:172.21.15.117:6827/1673738625] conn(0x7fe2fc074b70 0x7fe2fc077010 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:28.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.310+0000 7fe31f332640 20 client.16008.objecter put_session s=0x7fe2fc0744a0 osd=0 1 2024-05-15T13:20:28.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.310+0000 7fe31f332640 20 client.16008.objecter shutdown clearing up homeless session... 2024-05-15T13:20:28.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.310+0000 7fe31f332640 10 client.16008.objecter successfully canceled tick 2024-05-15T13:20:28.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.310+0000 7fe31f332640 1 -- 172.21.15.117:0/1001050348 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fe2fc06ce60 msgr2=0x7fe2fc06f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:28.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.310+0000 7fe31f332640 1 --2- 172.21.15.117:0/1001050348 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fe2fc06ce60 0x7fe2fc06f320 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fe308036000 tx=0x7fe308008030 comp rx=0 tx=0).stop 2024-05-15T13:20:28.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.310+0000 7fe31f332640 10 monclient: shutdown 2024-05-15T13:20:28.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.310+0000 7fe31f332640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:20:28.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.310+0000 7fe31f332640 1 -- 172.21.15.117:0/1001050348 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe31806b010 msgr2=0x7fe3180ae0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:28.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.310+0000 7fe31f332640 1 --2- 172.21.15.117:0/1001050348 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe31806b010 0x7fe3180ae0b0 secure :-1 s=READY pgs=3293 cs=0 l=1 rev1=1 crypto rx=0x7fe31400c6b0 tx=0x7fe31400cb70 comp rx=0 tx=0).stop 2024-05-15T13:20:28.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.310+0000 7fe31d8cf640 1 -- 172.21.15.117:0/1001050348 reap_dead start 2024-05-15T13:20:28.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.310+0000 7fe31f332640 1 -- 172.21.15.117:0/1001050348 shutdown_connections 2024-05-15T13:20:28.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.310+0000 7fe31f332640 1 -- 172.21.15.117:0/1001050348 >> 172.21.15.117:0/1001050348 conn(0x7fe3180b19b0 msgr2=0x7fe31813a650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:20:28.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.310+0000 7fe31f332640 1 -- 172.21.15.117:0/1001050348 shutdown_connections 2024-05-15T13:20:28.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.310+0000 7fe31f332640 1 -- 172.21.15.117:0/1001050348 wait complete. 2024-05-15T13:20:28.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.310+0000 7fe31f332640 1 librados: shutdown 2024-05-15T13:20:28.320 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-05-15T13:20:28.423 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.424+0000 7f7b0f956640 10 monclient: get_monmap_and_config 2024-05-15T13:20:28.423 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.424+0000 7f7b0f956640 10 monclient: build_initial_monmap 2024-05-15T13:20:28.423 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.424+0000 7f7b0f956640 10 monclient: monmap: 2024-05-15T13:20:28.423 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T13:20:28.423 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:28.423 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T13:20:28.425907+0000 2024-05-15T13:20:28.423 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T13:20:28.425907+0000 2024-05-15T13:20:28.424 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T13:20:28.424 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:28.424 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T13:20:28.424 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T13:20:28.424 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T13:20:28.424 INFO:teuthology.orchestra.run.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T13:20:28.424 INFO:teuthology.orchestra.run.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T13:20:28.424 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:28.426 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.425+0000 7f7b0f956640 1 Processor -- start 2024-05-15T13:20:28.426 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.425+0000 7f7b0f956640 1 -- start start 2024-05-15T13:20:28.426 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.425+0000 7f7b0f956640 10 monclient: init 2024-05-15T13:20:28.426 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.426+0000 7f7b0f956640 10 monclient: _reopen_session rank -1 2024-05-15T13:20:28.426 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.426+0000 7f7b0f956640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-05-15T13:20:28.426 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.426+0000 7f7b0f956640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7b080b4d00 addr v1:172.21.15.117:6789/0 2024-05-15T13:20:28.426 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.426+0000 7f7b0f956640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f7b080b50e0 0x7f7b080b1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:28.427 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.426+0000 7f7b0f956640 10 monclient(hunting): picked mon.noname-c con 0x7f7b080b50e0 addr v2:172.21.15.178:3300/0 2024-05-15T13:20:28.427 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.426+0000 7f7b0f956640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7b080a94b0 0x7f7b080b0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:28.427 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.426+0000 7f7b0f956640 10 monclient(hunting): picked mon.noname-b con 0x7f7b080a94b0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T13:20:28.427 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.426+0000 7f7b0f956640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b080ab1b0 con 0x7f7b080b4d00 2024-05-15T13:20:28.427 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.426+0000 7f7b0f956640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:28.427 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.426+0000 7f7b0f956640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b080b23c0 con 0x7f7b080b50e0 2024-05-15T13:20:28.427 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.426+0000 7f7b0f956640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:28.427 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.426+0000 7f7b0f956640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b080b1520 con 0x7f7b080a94b0 2024-05-15T13:20:28.427 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.426+0000 7f7b0f956640 10 monclient(hunting): _renew_subs 2024-05-15T13:20:28.427 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.426+0000 7f7b0f956640 10 monclient(hunting): authenticate will time out at 3591.668111s 2024-05-15T13:20:28.427 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.426+0000 7f7b0def3640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7b080a94b0 0x7f7b080b0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:28.427 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.426+0000 7f7b0cef1640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f7b080b50e0 0x7f7b080b1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:28.427 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.427+0000 7f7b0cef1640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f7b080b50e0 0x7f7b080b1e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:36036/0 (socket says 172.21.15.117:36036) 2024-05-15T13:20:28.427 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.427+0000 7f7b0cef1640 1 -- 172.21.15.117:0/2563372100 learned_addr learned my addr 172.21.15.117:0/2563372100 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T13:20:28.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.427+0000 7f7b0cef1640 10 monclient(hunting): get_auth_request con 0x7f7b080b50e0 auth_method 0 2024-05-15T13:20:28.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.427+0000 7f7b0cef1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:28.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.427+0000 7f7b0cef1640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:28.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.427+0000 7f7b0cef1640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:28.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.427+0000 7f7b0def3640 10 monclient(hunting): get_auth_request con 0x7f7b080a94b0 auth_method 0 2024-05-15T13:20:28.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.427+0000 7f7b0def3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:28.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.427+0000 7f7b0def3640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:28.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.427+0000 7f7b0def3640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:28.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.427+0000 7f7b0cef1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:28.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.427+0000 7f7b0cef1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:28.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.427+0000 7f7b0cef1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:28.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.427+0000 7f7b0def3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:28.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.427+0000 7f7b0def3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:28.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.427+0000 7f7b0def3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:28.429 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.429+0000 7f7b0cef1640 10 monclient(hunting): handle_auth_done global_id 22141 payload 274 2024-05-15T13:20:28.429 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.429+0000 7f7b0cef1640 1 -- 172.21.15.117:0/2563372100 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7b080a94b0 msgr2=0x7f7b080b0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T13:20:28.429 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.429+0000 7f7b0cef1640 1 --2- 172.21.15.117:0/2563372100 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7b080a94b0 0x7f7b080b0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:28.429 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.429+0000 7f7b0cef1640 1 -- 172.21.15.117:0/2563372100 >> v1:172.21.15.117:6789/0 conn(0x7f7b080b4d00 legacy=0x7f7b08145c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T13:20:28.429 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.429+0000 7f7b0cef1640 10 monclient: _finish_hunting 0 2024-05-15T13:20:28.429 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.429+0000 7f7b0cef1640 1 monclient: found mon.noname-c 2024-05-15T13:20:28.429 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.429+0000 7f7b0cef1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:20:28.429 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.429+0000 7f7b0cef1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.178:3300/0 2024-05-15T13:20:28.429 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.429+0000 7f7b0cef1640 1 -- 172.21.15.117:0/2563372100 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b080b16d0 con 0x7f7b080b50e0 2024-05-15T13:20:28.430 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.431+0000 7f7b0cef1640 1 --2- 172.21.15.117:0/2563372100 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7b080b50e0 0x7f7b080b1e80 secure :-1 s=READY pgs=5441 cs=0 l=1 rev1=1 crypto rx=0x7f7af8009870 tx=0x7f7af802b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd07bba8d9472d10 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:28.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.435+0000 7f7afffff640 1 -- 172.21.15.117:0/2563372100 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af80317c0 con 0x7f7b080b50e0 2024-05-15T13:20:28.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.435+0000 7f7afffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:28.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.435+0000 7f7afffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T13:20:28.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.435+0000 7f7afffff640 10 monclient: dump: 2024-05-15T13:20:28.435 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:28.435 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:28.436 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:28.436 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:28.436 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:28.436 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:28.436 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:28.436 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:28.436 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:28.436 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:28.436 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.435+0000 7f7afffff640 10 monclient: _finish_auth 0 2024-05-15T13:20:28.436 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.435+0000 7f7afffff640 10 monclient: _check_auth_tickets 2024-05-15T13:20:28.436 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.435+0000 7f7afffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:20:28.436 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.435+0000 7f7b0f956640 5 monclient: authenticate success, global_id 22141 2024-05-15T13:20:28.436 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.435+0000 7f7afffff640 1 -- 172.21.15.117:0/2563372100 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7af8037070 con 0x7f7b080b50e0 2024-05-15T13:20:28.436 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.435+0000 7f7b0f956640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T13:20:28.437 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.435+0000 7f7afffff640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:20:28.437 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.435+0000 7f7b0f956640 10 monclient: get_monmap_and_config success 2024-05-15T13:20:28.437 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.435+0000 7f7b0f956640 4 set_mon_vals no callback set 2024-05-15T13:20:28.437 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.435+0000 7f7afffff640 1 -- 172.21.15.117:0/2563372100 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af8031b80 con 0x7f7b080b50e0 2024-05-15T13:20:28.437 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.435+0000 7f7afffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:28.437 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.435+0000 7f7afffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T13:20:28.437 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.435+0000 7f7afffff640 10 monclient: dump: 2024-05-15T13:20:28.437 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:28.437 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:28.437 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:28.437 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:28.437 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:28.437 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:28.437 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:28.438 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:28.438 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:28.438 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:28.438 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.435+0000 7f7b0f956640 10 monclient: shutdown 2024-05-15T13:20:28.438 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.435+0000 7f7b0f956640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:20:28.438 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.435+0000 7f7b0f956640 1 -- 172.21.15.117:0/2563372100 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7b080b50e0 msgr2=0x7f7b080b1e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:28.438 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.435+0000 7f7b0f956640 1 --2- 172.21.15.117:0/2563372100 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7b080b50e0 0x7f7b080b1e80 secure :-1 s=READY pgs=5441 cs=0 l=1 rev1=1 crypto rx=0x7f7af8009870 tx=0x7f7af802b860 comp rx=0 tx=0).stop 2024-05-15T13:20:28.438 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.436+0000 7f7b0f956640 1 -- 172.21.15.117:0/2563372100 shutdown_connections 2024-05-15T13:20:28.438 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.436+0000 7f7b0f956640 1 --2- 172.21.15.117:0/2563372100 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7b080a94b0 0x7f7b080b0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:28.438 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.436+0000 7f7b0f956640 1 --2- 172.21.15.117:0/2563372100 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7b080b50e0 0x7f7b080b1e80 unknown :-1 s=CLOSED pgs=5441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:28.438 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.436+0000 7f7b0f956640 1 -- 172.21.15.117:0/2563372100 >> 172.21.15.117:0/2563372100 conn(0x7f7b08066cc0 msgr2=0x7f7b080670d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:20:28.438 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.436+0000 7f7b0f956640 1 -- 172.21.15.117:0/2563372100 shutdown_connections 2024-05-15T13:20:28.438 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.436+0000 7f7b0f956640 1 -- 172.21.15.117:0/2563372100 wait complete. 2024-05-15T13:20:28.438 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.437+0000 7f7b0f956640 10 monclient: build_initial_monmap 2024-05-15T13:20:28.439 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.437+0000 7f7b0f956640 10 monclient: monmap: 2024-05-15T13:20:28.439 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T13:20:28.439 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T13:20:28.439 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-05-15T13:20:28.439 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-05-15T13:20:28.439 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T13:20:28.439 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:28.439 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T13:20:28.439 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T13:20:28.439 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T13:20:28.439 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:28.442 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.437+0000 7f7b0f956640 1 librados: starting msgr at 2024-05-15T13:20:28.442 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.437+0000 7f7b0f956640 1 librados: starting objecter 2024-05-15T13:20:28.442 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.437+0000 7f7b0f956640 1 Processor -- start 2024-05-15T13:20:28.442 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.437+0000 7f7b0f956640 1 -- start start 2024-05-15T13:20:28.442 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.437+0000 7f7b0f956640 1 librados: setting wanted keys 2024-05-15T13:20:28.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.437+0000 7f7b0f956640 1 librados: calling monclient init 2024-05-15T13:20:28.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.437+0000 7f7b0f956640 10 monclient: init 2024-05-15T13:20:28.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.438+0000 7f7b0f956640 10 monclient: _reopen_session rank -1 2024-05-15T13:20:28.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.438+0000 7f7b0f956640 10 monclient: _add_conns ranks=[0,1,2] 2024-05-15T13:20:28.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.438+0000 7f7b0f956640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7b080a94b0 0x7f7b081381f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:28.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.438+0000 7f7b0f956640 10 monclient(hunting): picked mon.noname-a con 0x7f7b080a94b0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T13:20:28.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.438+0000 7f7b0f956640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7b080b4d00 0x7f7b08144170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:28.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.438+0000 7f7b0f956640 10 monclient(hunting): picked mon.noname-b con 0x7f7b080b4d00 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T13:20:28.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.438+0000 7f7b0f956640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7b081446b0 0x7f7b081db240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:28.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.438+0000 7f7b0f956640 10 monclient(hunting): picked mon.noname-c con 0x7f7b081446b0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T13:20:28.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.438+0000 7f7b0f956640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:28.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.438+0000 7f7b0f956640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0814d130 con 0x7f7b080a94b0 2024-05-15T13:20:28.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.438+0000 7f7b0f956640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:28.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.438+0000 7f7b0f956640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0814cfb0 con 0x7f7b080b4d00 2024-05-15T13:20:28.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.438+0000 7f7b0f956640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:28.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.438+0000 7f7b0f956640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0814d2b0 con 0x7f7b081446b0 2024-05-15T13:20:28.444 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.438+0000 7f7b0f956640 10 monclient(hunting): _renew_subs 2024-05-15T13:20:28.444 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.438+0000 7f7b0f956640 10 monclient(hunting): authenticate will time out at 3591.679931s 2024-05-15T13:20:28.444 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.438+0000 7f7b0cef1640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7b080b4d00 0x7f7b08144170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:28.444 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.439+0000 7f7b0cef1640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7b080b4d00 0x7f7b08144170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:36044/0 (socket says 172.21.15.117:36044) 2024-05-15T13:20:28.444 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.439+0000 7f7b0cef1640 1 -- 172.21.15.117:0/2083500910 learned_addr learned my addr 172.21.15.117:0/2083500910 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T13:20:28.444 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.439+0000 7f7b0cef1640 10 monclient(hunting): get_auth_request con 0x7f7b080b4d00 auth_method 0 2024-05-15T13:20:28.444 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.439+0000 7f7b0cef1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:28.444 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.439+0000 7f7b0cef1640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:28.444 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.439+0000 7f7b0cef1640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:28.444 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.439+0000 7f7b0cef1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:28.444 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.439+0000 7f7b0cef1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:28.444 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.439+0000 7f7b0cef1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:28.444 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.439+0000 7f7b0d6f2640 1 --2- 172.21.15.117:0/2083500910 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7b080a94b0 0x7f7b081381f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:28.444 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.439+0000 7f7b0d6f2640 10 monclient(hunting): get_auth_request con 0x7f7b080a94b0 auth_method 0 2024-05-15T13:20:28.444 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.439+0000 7f7b0d6f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:28.444 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.439+0000 7f7b0d6f2640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:28.445 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.439+0000 7f7b0d6f2640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:28.445 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.440+0000 7f7b0def3640 1 --2- 172.21.15.117:0/2083500910 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7b081446b0 0x7f7b081db240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:28.445 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.440+0000 7f7b0d6f2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:28.445 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.440+0000 7f7b0d6f2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:28.445 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.440+0000 7f7b0d6f2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:28.445 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.442+0000 7f7b0def3640 10 monclient(hunting): get_auth_request con 0x7f7b081446b0 auth_method 0 2024-05-15T13:20:28.445 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.442+0000 7f7b0def3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:28.445 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.442+0000 7f7b0def3640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:28.445 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.442+0000 7f7b0def3640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:28.445 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.442+0000 7f7b0cef1640 10 monclient(hunting): handle_auth_done global_id 22144 payload 824 2024-05-15T13:20:28.445 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.442+0000 7f7b0cef1640 1 -- 172.21.15.117:0/2083500910 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7b081446b0 msgr2=0x7f7b081db240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:28.445 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.442+0000 7f7b0cef1640 1 --2- 172.21.15.117:0/2083500910 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7b081446b0 0x7f7b081db240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:28.445 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.442+0000 7f7b0cef1640 1 -- 172.21.15.117:0/2083500910 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7b080a94b0 msgr2=0x7f7b081381f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:28.445 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.442+0000 7f7b0cef1640 1 --2- 172.21.15.117:0/2083500910 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7b080a94b0 0x7f7b081381f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:28.445 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.442+0000 7f7b0cef1640 10 monclient: _finish_hunting 0 2024-05-15T13:20:28.445 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.442+0000 7f7b0cef1640 1 monclient: found mon.noname-b 2024-05-15T13:20:28.446 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.442+0000 7f7b0cef1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:20:28.446 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.442+0000 7f7b0cef1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2024-05-15T13:20:28.446 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.442+0000 7f7b0cef1640 1 -- 172.21.15.117:0/2083500910 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b08141040 con 0x7f7b080b4d00 2024-05-15T13:20:28.447 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.444+0000 7f7b0cef1640 1 --2- 172.21.15.117:0/2083500910 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7b080b4d00 0x7f7b08144170 secure :-1 s=READY pgs=5442 cs=0 l=1 rev1=1 crypto rx=0x7f7af8009870 tx=0x7f7af80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:28.447 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.444+0000 7f7afdffb640 10 client.?.objecter ms_handle_connect 0x7f7b080b4d00 2024-05-15T13:20:28.447 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.444+0000 7f7afdffb640 10 client.?.objecter resend_mon_ops 2024-05-15T13:20:28.447 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.448+0000 7f7afdffb640 1 -- 172.21.15.117:0/2083500910 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af8032650 con 0x7f7b080b4d00 2024-05-15T13:20:28.447 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.448+0000 7f7afdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:28.447 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.448+0000 7f7afdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T13:20:28.447 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.448+0000 7f7afdffb640 10 monclient: dump: 2024-05-15T13:20:28.448 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:28.448 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:28.448 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:28.448 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:28.448 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:28.448 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:28.448 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:28.448 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:28.448 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:28.448 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:28.448 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.448+0000 7f7afdffb640 10 monclient: _finish_auth 0 2024-05-15T13:20:28.448 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.448+0000 7f7afdffb640 10 monclient: _check_auth_tickets 2024-05-15T13:20:28.448 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.448+0000 7f7afdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:20:28.448 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.448+0000 7f7afdffb640 1 -- 172.21.15.117:0/2083500910 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7af80327f0 con 0x7f7b080b4d00 2024-05-15T13:20:28.448 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.448+0000 7f7b0f956640 5 monclient: authenticate success, global_id 22144 2024-05-15T13:20:28.449 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.448+0000 7f7afdffb640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:20:28.449 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.448+0000 7f7afdffb640 1 -- 172.21.15.117:0/2083500910 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af8032be0 con 0x7f7b080b4d00 2024-05-15T13:20:28.449 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.448+0000 7f7b0f956640 10 monclient: _renew_subs 2024-05-15T13:20:28.449 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.448+0000 7f7b0f956640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T13:20:28.449 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.448+0000 7f7b0f956640 1 -- 172.21.15.117:0/2083500910 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b08141320 con 0x7f7b080b4d00 2024-05-15T13:20:28.449 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.448+0000 7f7afdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:28.449 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.448+0000 7f7afdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T13:20:28.449 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.448+0000 7f7afdffb640 10 monclient: dump: 2024-05-15T13:20:28.449 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:28.449 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:28.449 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:28.449 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:28.449 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:28.449 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:28.449 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:28.450 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:28.450 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:28.450 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:28.450 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.448+0000 7f7aff7fe640 4 set_mon_vals no callback set 2024-05-15T13:20:28.450 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.448+0000 7f7b0f956640 10 client.22144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:20:28.450 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.448+0000 7f7b0f956640 10 monclient: _renew_subs 2024-05-15T13:20:28.450 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.448+0000 7f7b0f956640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T13:20:28.450 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.448+0000 7f7b0f956640 1 -- 172.21.15.117:0/2083500910 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b08138730 con 0x7f7b080b4d00 2024-05-15T13:20:28.450 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.448+0000 7f7b0f956640 1 librados: init done 2024-05-15T13:20:28.451 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.449+0000 7f7b0f956640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T13:20:28.451 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.449+0000 7f7b0f956640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:20:28.451 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.449+0000 7f7b0f956640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T13:20:28.451 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.449+0000 7f7b0f956640 1 -- 172.21.15.117:0/2083500910 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b0804f0e0 con 0x7f7b080b4d00 2024-05-15T13:20:28.694 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.695+0000 7f7afdffb640 1 -- 172.21.15.117:0/2083500910 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f7af804c060 con 0x7f7b080b4d00 2024-05-15T13:20:28.694 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.695+0000 7f7afdffb640 1 --2- 172.21.15.117:0/2083500910 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f7aec06ce60 0x7f7aec06f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:28.694 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.696+0000 7f7b0d6f2640 1 --2- 172.21.15.117:0/2083500910 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f7aec06ce60 0x7f7aec06f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:28.694 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.696+0000 7f7b0d6f2640 10 monclient: get_auth_request con 0x7f7aec06ce60 auth_method 0 2024-05-15T13:20:28.695 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.696+0000 7f7afdffb640 10 client.22144.objecter ms_handle_connect 0x7f7aec06ce60 2024-05-15T13:20:28.695 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.696+0000 7f7b0d6f2640 1 --2- 172.21.15.117:0/2083500910 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f7aec06ce60 0x7f7aec06f320 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f7b040017f0 tx=0x7f7b04009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:28.697 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.699+0000 7f7afdffb640 1 -- 172.21.15.117:0/2083500910 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(1225..1225 src has 912..1225) v4 ==== 6365+0+0 (secure 0 0 0) 0x7f7af8078340 con 0x7f7b080b4d00 2024-05-15T13:20:28.697 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.699+0000 7f7afdffb640 10 client.22144.objecter ms_dispatch 0x7f7b08000b90 osd_map(1225..1225 src has 912..1225) v4 2024-05-15T13:20:28.697 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.699+0000 7f7afdffb640 3 client.22144.objecter handle_osd_map got epochs [1225,1225] > 0 2024-05-15T13:20:28.698 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.699+0000 7f7afdffb640 3 client.22144.objecter handle_osd_map decoding full epoch 1225 2024-05-15T13:20:28.698 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.699+0000 7f7afdffb640 20 client.22144.objecter dump_active .. 0 homeless 2024-05-15T13:20:28.709 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.708+0000 7f7afdffb640 1 -- 172.21.15.117:0/2083500910 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f7af8040070 con 0x7f7b080b4d00 2024-05-15T13:20:28.709 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.708+0000 7f7afdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:20:28.709 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.708+0000 7f7afdffb640 10 monclient: _finish_command 1 = system:0 2024-05-15T13:20:28.890 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.891+0000 7f7b0f956640 1 -- 172.21.15.117:0/2083500910 --> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f7b08067d90 con 0x7f7aec06ce60 2024-05-15T13:20:28.890 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.891+0000 7f7afdffb640 1 -- 172.21.15.117:0/2083500910 <== mgr.4107 v2:172.21.15.117:6800/1785766523 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f7b08067d90 con 0x7f7aec06ce60 2024-05-15T13:20:28.890 INFO:teuthology.orchestra.run.smithi117.stderr:instructed osd(s) 0 to deep-scrub 2024-05-15T13:20:28.893 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.893+0000 7f7b0f956640 10 librados: watch_flush enter 2024-05-15T13:20:28.893 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.893+0000 7f7b0f956640 10 librados: watch_flush exit 2024-05-15T13:20:28.893 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.893+0000 7f7b0f956640 20 client.22144.objecter shutdown clearing up homeless session... 2024-05-15T13:20:28.893 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.893+0000 7f7b0f956640 10 client.22144.objecter successfully canceled tick 2024-05-15T13:20:28.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.894+0000 7f7b0f956640 1 -- 172.21.15.117:0/2083500910 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f7aec06ce60 msgr2=0x7f7aec06f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:28.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.894+0000 7f7b0f956640 1 --2- 172.21.15.117:0/2083500910 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f7aec06ce60 0x7f7aec06f320 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f7b040017f0 tx=0x7f7b04009040 comp rx=0 tx=0).stop 2024-05-15T13:20:28.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.894+0000 7f7b0f956640 10 monclient: shutdown 2024-05-15T13:20:28.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.894+0000 7f7b0f956640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:20:28.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.894+0000 7f7b0f956640 1 -- 172.21.15.117:0/2083500910 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7b080b4d00 msgr2=0x7f7b08144170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:28.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.894+0000 7f7b0f956640 1 --2- 172.21.15.117:0/2083500910 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7b080b4d00 0x7f7b08144170 secure :-1 s=READY pgs=5442 cs=0 l=1 rev1=1 crypto rx=0x7f7af8009870 tx=0x7f7af80046c0 comp rx=0 tx=0).stop 2024-05-15T13:20:28.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.894+0000 7f7b0f956640 1 -- 172.21.15.117:0/2083500910 shutdown_connections 2024-05-15T13:20:28.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.894+0000 7f7b0f956640 1 --2- 172.21.15.117:0/2083500910 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7b081446b0 0x7f7b081db240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:28.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.894+0000 7f7b0f956640 1 --2- 172.21.15.117:0/2083500910 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f7aec06ce60 0x7f7aec06f320 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:28.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.894+0000 7f7b0f956640 1 --2- 172.21.15.117:0/2083500910 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7b080b4d00 0x7f7b08144170 unknown :-1 s=CLOSED pgs=5442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:28.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.894+0000 7f7b0f956640 1 --2- 172.21.15.117:0/2083500910 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7b080a94b0 0x7f7b081381f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:28.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.894+0000 7f7b0f956640 1 -- 172.21.15.117:0/2083500910 >> 172.21.15.117:0/2083500910 conn(0x7f7b08066cc0 msgr2=0x7f7b080b86c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:20:28.895 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.894+0000 7f7b0f956640 1 -- 172.21.15.117:0/2083500910 shutdown_connections 2024-05-15T13:20:28.895 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.894+0000 7f7b0f956640 1 -- 172.21.15.117:0/2083500910 wait complete. 2024-05-15T13:20:28.895 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:28.894+0000 7f7b0f956640 1 librados: shutdown 2024-05-15T13:20:28.904 INFO:tasks.ceph:Scrubbing osd.1 2024-05-15T13:20:28.904 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-05-15T13:20:29.006 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.007+0000 7fd5d2c30640 10 monclient: get_monmap_and_config 2024-05-15T13:20:29.006 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.007+0000 7fd5d2c30640 10 monclient: build_initial_monmap 2024-05-15T13:20:29.006 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.007+0000 7fd5d2c30640 10 monclient: monmap: 2024-05-15T13:20:29.006 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T13:20:29.006 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:29.006 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T13:20:29.008768+0000 2024-05-15T13:20:29.007 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T13:20:29.008768+0000 2024-05-15T13:20:29.007 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T13:20:29.007 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:29.007 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T13:20:29.007 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T13:20:29.007 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T13:20:29.007 INFO:teuthology.orchestra.run.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T13:20:29.007 INFO:teuthology.orchestra.run.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T13:20:29.007 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:29.010 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.008+0000 7fd5d2c30640 1 Processor -- start 2024-05-15T13:20:29.010 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.008+0000 7fd5d2c30640 1 -- start start 2024-05-15T13:20:29.010 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.008+0000 7fd5d2c30640 10 monclient: init 2024-05-15T13:20:29.010 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.009+0000 7fd5d2c30640 10 monclient: _reopen_session rank -1 2024-05-15T13:20:29.011 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.009+0000 7fd5d2c30640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-05-15T13:20:29.011 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.009+0000 7fd5d2c30640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd5cc070bc0 addr v1:172.21.15.117:6789/0 2024-05-15T13:20:29.011 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.009+0000 7fd5d2c30640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fd5cc0b07e0 0x7fd5cc13d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:29.011 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.009+0000 7fd5d2c30640 10 monclient(hunting): picked mon.noname-a con 0x7fd5cc0b07e0 addr v2:172.21.15.117:3300/0 2024-05-15T13:20:29.011 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.009+0000 7fd5d2c30640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd5cc070fa0 0x7fd5cc13de20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:29.011 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.009+0000 7fd5d2c30640 10 monclient(hunting): picked mon.noname-b con 0x7fd5cc070fa0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T13:20:29.011 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.009+0000 7fd5d2c30640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5cc1448a0 con 0x7fd5cc070bc0 2024-05-15T13:20:29.011 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.009+0000 7fd5d2c30640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:29.012 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.009+0000 7fd5d2c30640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5cc072b30 con 0x7fd5cc0b07e0 2024-05-15T13:20:29.012 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.009+0000 7fd5d2c30640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:29.012 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.009+0000 7fd5d2c30640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5cc0a9550 con 0x7fd5cc070fa0 2024-05-15T13:20:29.012 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.009+0000 7fd5d2c30640 10 monclient(hunting): _renew_subs 2024-05-15T13:20:29.012 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.009+0000 7fd5d2c30640 10 monclient(hunting): authenticate will time out at 3592.250920s 2024-05-15T13:20:29.013 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.009+0000 7fd5d11cd640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd5cc070fa0 0x7fd5cc13de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:29.013 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.009+0000 7fd5d09cc640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fd5cc070bc0 0x7fd5cc144190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:54102/0 (socket says 172.21.15.117:54102) 2024-05-15T13:20:29.013 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.009+0000 7fd5d09cc640 1 -- 172.21.15.117:0/82589314 learned_addr learned my addr 172.21.15.117:0/82589314 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-05-15T13:20:29.013 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.010+0000 7fd5cbfff640 1 --2- 172.21.15.117:0/82589314 >> v2:172.21.15.117:3300/0 conn(0x7fd5cc0b07e0 0x7fd5cc13d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:29.013 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.010+0000 7fd5d11cd640 10 monclient(hunting): get_auth_request con 0x7fd5cc070fa0 auth_method 0 2024-05-15T13:20:29.013 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.010+0000 7fd5d11cd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:29.013 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.010+0000 7fd5d11cd640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:29.014 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.010+0000 7fd5d11cd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:29.014 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.010+0000 7fd5d11cd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:29.014 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.010+0000 7fd5d11cd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:29.014 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.010+0000 7fd5d11cd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:29.014 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.010+0000 7fd5cbfff640 10 monclient(hunting): get_auth_request con 0x7fd5cc0b07e0 auth_method 0 2024-05-15T13:20:29.014 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.010+0000 7fd5cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:29.015 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.010+0000 7fd5cbfff640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:29.015 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.010+0000 7fd5cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:29.015 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.011+0000 7fd5cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:29.015 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.011+0000 7fd5cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:29.015 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.011+0000 7fd5cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:29.016 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.012+0000 7fd5d11cd640 10 monclient(hunting): handle_auth_done global_id 18728 payload 274 2024-05-15T13:20:29.016 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.012+0000 7fd5d11cd640 1 -- 172.21.15.117:0/82589314 >> v2:172.21.15.117:3300/0 conn(0x7fd5cc0b07e0 msgr2=0x7fd5cc13d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T13:20:29.016 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.012+0000 7fd5d11cd640 1 --2- 172.21.15.117:0/82589314 >> v2:172.21.15.117:3300/0 conn(0x7fd5cc0b07e0 0x7fd5cc13d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:29.016 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.012+0000 7fd5d11cd640 1 -- 172.21.15.117:0/82589314 >> v1:172.21.15.117:6789/0 conn(0x7fd5cc070bc0 legacy=0x7fd5cc144190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:29.016 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.012+0000 7fd5d11cd640 10 monclient: _finish_hunting 0 2024-05-15T13:20:29.017 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.012+0000 7fd5d11cd640 1 monclient: found mon.noname-b 2024-05-15T13:20:29.017 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.012+0000 7fd5d11cd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:20:29.017 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.012+0000 7fd5d11cd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.117:3301/0 2024-05-15T13:20:29.017 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.012+0000 7fd5d11cd640 1 -- 172.21.15.117:0/82589314 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5cc071fa0 con 0x7fd5cc070fa0 2024-05-15T13:20:29.017 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.014+0000 7fd5d11cd640 1 --2- 172.21.15.117:0/82589314 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd5cc070fa0 0x7fd5cc13de20 secure :-1 s=READY pgs=3381 cs=0 l=1 rev1=1 crypto rx=0x7fd5c4009d90 tx=0x7fd5c402be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93c99c756359bd9a server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:29.023 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.018+0000 7fd5cb7fe640 1 -- 172.21.15.117:0/82589314 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5c4039070 con 0x7fd5cc070fa0 2024-05-15T13:20:29.023 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.018+0000 7fd5cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:29.023 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.018+0000 7fd5cb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T13:20:29.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.018+0000 7fd5cb7fe640 10 monclient: dump: 2024-05-15T13:20:29.024 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:29.024 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:29.024 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:29.024 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:29.024 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:29.024 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:29.024 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:29.025 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:29.025 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:29.025 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:29.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.018+0000 7fd5cb7fe640 10 monclient: _finish_auth 0 2024-05-15T13:20:29.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.018+0000 7fd5cb7fe640 10 monclient: _check_auth_tickets 2024-05-15T13:20:29.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.018+0000 7fd5cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:20:29.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.018+0000 7fd5cb7fe640 1 -- 172.21.15.117:0/82589314 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5c402cbc0 con 0x7fd5cc070fa0 2024-05-15T13:20:29.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.018+0000 7fd5cb7fe640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:20:29.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.018+0000 7fd5cb7fe640 1 -- 172.21.15.117:0/82589314 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5c4034440 con 0x7fd5cc070fa0 2024-05-15T13:20:29.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.018+0000 7fd5d2c30640 5 monclient: authenticate success, global_id 18728 2024-05-15T13:20:29.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.018+0000 7fd5d2c30640 10 monclient: get_monmap_and_config success 2024-05-15T13:20:29.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.018+0000 7fd5d2c30640 4 set_mon_vals no callback set 2024-05-15T13:20:29.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.018+0000 7fd5cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:29.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.018+0000 7fd5cb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T13:20:29.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.018+0000 7fd5cb7fe640 10 monclient: dump: 2024-05-15T13:20:29.027 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:29.027 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:29.027 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:29.027 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:29.027 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:29.027 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:29.027 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:29.027 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:29.028 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:29.028 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:29.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.019+0000 7fd5d2c30640 10 monclient: shutdown 2024-05-15T13:20:29.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.019+0000 7fd5d2c30640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:20:29.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.019+0000 7fd5d2c30640 1 -- 172.21.15.117:0/82589314 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd5cc070fa0 msgr2=0x7fd5cc13de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:29.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.019+0000 7fd5d2c30640 1 --2- 172.21.15.117:0/82589314 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd5cc070fa0 0x7fd5cc13de20 secure :-1 s=READY pgs=3381 cs=0 l=1 rev1=1 crypto rx=0x7fd5c4009d90 tx=0x7fd5c402be10 comp rx=0 tx=0).stop 2024-05-15T13:20:29.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.019+0000 7fd5d2c30640 1 -- 172.21.15.117:0/82589314 shutdown_connections 2024-05-15T13:20:29.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.019+0000 7fd5d2c30640 1 --2- 172.21.15.117:0/82589314 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd5cc070fa0 0x7fd5cc13de20 unknown :-1 s=CLOSED pgs=3381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:29.029 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.019+0000 7fd5d2c30640 1 --2- 172.21.15.117:0/82589314 >> v2:172.21.15.117:3300/0 conn(0x7fd5cc0b07e0 0x7fd5cc13d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:29.029 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.019+0000 7fd5d2c30640 1 -- 172.21.15.117:0/82589314 >> 172.21.15.117:0/82589314 conn(0x7fd5cc0b2ac0 msgr2=0x7fd5cc0b2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:20:29.029 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.019+0000 7fd5d2c30640 1 -- 172.21.15.117:0/82589314 shutdown_connections 2024-05-15T13:20:29.029 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.019+0000 7fd5d2c30640 1 -- 172.21.15.117:0/82589314 wait complete. 2024-05-15T13:20:29.029 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.020+0000 7fd5d2c30640 10 monclient: build_initial_monmap 2024-05-15T13:20:29.029 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.020+0000 7fd5d2c30640 10 monclient: monmap: 2024-05-15T13:20:29.029 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T13:20:29.029 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T13:20:29.030 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-05-15T13:20:29.030 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-05-15T13:20:29.030 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T13:20:29.030 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:29.030 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T13:20:29.030 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T13:20:29.030 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T13:20:29.030 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:29.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.020+0000 7fd5d2c30640 1 librados: starting msgr at 2024-05-15T13:20:29.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.020+0000 7fd5d2c30640 1 librados: starting objecter 2024-05-15T13:20:29.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.020+0000 7fd5d2c30640 1 Processor -- start 2024-05-15T13:20:29.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.020+0000 7fd5d2c30640 1 -- start start 2024-05-15T13:20:29.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.020+0000 7fd5d2c30640 1 librados: setting wanted keys 2024-05-15T13:20:29.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.020+0000 7fd5d2c30640 1 librados: calling monclient init 2024-05-15T13:20:29.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.020+0000 7fd5d2c30640 10 monclient: init 2024-05-15T13:20:29.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.021+0000 7fd5d2c30640 10 monclient: _reopen_session rank -1 2024-05-15T13:20:29.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.021+0000 7fd5d2c30640 10 monclient: _add_conns ranks=[0,2,1] 2024-05-15T13:20:29.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.021+0000 7fd5d2c30640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5cc070bc0 0x7fd5cc1337d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:29.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.021+0000 7fd5d2c30640 10 monclient(hunting): picked mon.noname-a con 0x7fd5cc070bc0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T13:20:29.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.021+0000 7fd5d2c30640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd5cc070fa0 0x7fd5cc133d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:29.032 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.021+0000 7fd5d2c30640 10 monclient(hunting): picked mon.noname-c con 0x7fd5cc070fa0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T13:20:29.032 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.021+0000 7fd5d2c30640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd5cc0b07e0 0x7fd5cc1d7c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:29.032 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.021+0000 7fd5d2c30640 10 monclient(hunting): picked mon.noname-b con 0x7fd5cc0b07e0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T13:20:29.032 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.021+0000 7fd5d2c30640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:29.032 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.021+0000 7fd5d2c30640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5cc14b1f0 con 0x7fd5cc070bc0 2024-05-15T13:20:29.032 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.021+0000 7fd5d2c30640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:29.032 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.021+0000 7fd5d2c30640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5cc14b070 con 0x7fd5cc0b07e0 2024-05-15T13:20:29.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.021+0000 7fd5d2c30640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:29.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.021+0000 7fd5d2c30640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5cc14b370 con 0x7fd5cc070fa0 2024-05-15T13:20:29.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.021+0000 7fd5d2c30640 10 monclient(hunting): _renew_subs 2024-05-15T13:20:29.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.021+0000 7fd5d2c30640 10 monclient(hunting): authenticate will time out at 3592.263209s 2024-05-15T13:20:29.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.021+0000 7fd5cbfff640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd5cc070fa0 0x7fd5cc133d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:29.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.021+0000 7fd5d09cc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5cc070bc0 0x7fd5cc1337d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:29.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.022+0000 7fd5d11cd640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd5cc0b07e0 0x7fd5cc1d7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:29.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.022+0000 7fd5d09cc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5cc070bc0 0x7fd5cc1337d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:54808/0 (socket says 172.21.15.117:54808) 2024-05-15T13:20:29.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.022+0000 7fd5d09cc640 1 -- 172.21.15.117:0/3658509248 learned_addr learned my addr 172.21.15.117:0/3658509248 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T13:20:29.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.022+0000 7fd5d09cc640 10 monclient(hunting): get_auth_request con 0x7fd5cc070bc0 auth_method 0 2024-05-15T13:20:29.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.022+0000 7fd5d09cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:29.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.022+0000 7fd5d09cc640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:29.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.022+0000 7fd5d09cc640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:29.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.022+0000 7fd5cbfff640 10 monclient(hunting): get_auth_request con 0x7fd5cc070fa0 auth_method 0 2024-05-15T13:20:29.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.022+0000 7fd5cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:29.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.022+0000 7fd5cbfff640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:29.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.022+0000 7fd5cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:29.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.022+0000 7fd5d11cd640 10 monclient(hunting): get_auth_request con 0x7fd5cc0b07e0 auth_method 0 2024-05-15T13:20:29.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.022+0000 7fd5d11cd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:29.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.022+0000 7fd5d11cd640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:29.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.022+0000 7fd5d11cd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:29.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.022+0000 7fd5d09cc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:29.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.022+0000 7fd5d09cc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:29.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.022+0000 7fd5d09cc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:29.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.022+0000 7fd5d11cd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:29.035 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.022+0000 7fd5d11cd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:29.035 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.022+0000 7fd5d11cd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:29.035 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.022+0000 7fd5cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:29.035 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.022+0000 7fd5cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:29.035 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.022+0000 7fd5cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:29.038 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.025+0000 7fd5d11cd640 10 monclient(hunting): handle_auth_done global_id 22147 payload 824 2024-05-15T13:20:29.038 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.025+0000 7fd5d11cd640 1 -- 172.21.15.117:0/3658509248 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd5cc070fa0 msgr2=0x7fd5cc133d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:29.039 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.025+0000 7fd5d11cd640 1 --2- 172.21.15.117:0/3658509248 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd5cc070fa0 0x7fd5cc133d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:29.039 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.025+0000 7fd5d11cd640 1 -- 172.21.15.117:0/3658509248 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5cc070bc0 msgr2=0x7fd5cc1337d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:29.039 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.025+0000 7fd5d11cd640 1 --2- 172.21.15.117:0/3658509248 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5cc070bc0 0x7fd5cc1337d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:29.039 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.025+0000 7fd5d11cd640 10 monclient: _finish_hunting 0 2024-05-15T13:20:29.039 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.025+0000 7fd5d11cd640 1 monclient: found mon.noname-b 2024-05-15T13:20:29.039 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.025+0000 7fd5d11cd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:20:29.040 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.025+0000 7fd5d11cd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2024-05-15T13:20:29.040 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.025+0000 7fd5d11cd640 1 -- 172.21.15.117:0/3658509248 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5cc135710 con 0x7fd5cc0b07e0 2024-05-15T13:20:29.040 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.027+0000 7fd5d11cd640 1 --2- 172.21.15.117:0/3658509248 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd5cc0b07e0 0x7fd5cc1d7c30 secure :-1 s=READY pgs=5443 cs=0 l=1 rev1=1 crypto rx=0x7fd5c4033000 tx=0x7fd5c4004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:29.040 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.027+0000 7fd5c97fa640 10 client.?.objecter ms_handle_connect 0x7fd5cc0b07e0 2024-05-15T13:20:29.040 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.027+0000 7fd5c97fa640 10 client.?.objecter resend_mon_ops 2024-05-15T13:20:29.040 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.031+0000 7fd5c97fa640 1 -- 172.21.15.117:0/3658509248 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5c4044020 con 0x7fd5cc0b07e0 2024-05-15T13:20:29.041 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.031+0000 7fd5c97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:29.041 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.031+0000 7fd5c97fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T13:20:29.041 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.031+0000 7fd5c97fa640 10 monclient: dump: 2024-05-15T13:20:29.041 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:29.041 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:29.041 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:29.042 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:29.042 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:29.042 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:29.042 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:29.042 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:29.042 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:29.042 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:29.042 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.031+0000 7fd5c97fa640 10 monclient: _finish_auth 0 2024-05-15T13:20:29.042 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.031+0000 7fd5c97fa640 10 monclient: _check_auth_tickets 2024-05-15T13:20:29.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.031+0000 7fd5c97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:20:29.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.031+0000 7fd5c97fa640 1 -- 172.21.15.117:0/3658509248 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5c403f470 con 0x7fd5cc0b07e0 2024-05-15T13:20:29.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.031+0000 7fd5d2c30640 5 monclient: authenticate success, global_id 22147 2024-05-15T13:20:29.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.031+0000 7fd5d2c30640 10 monclient: _renew_subs 2024-05-15T13:20:29.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.031+0000 7fd5d2c30640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T13:20:29.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.031+0000 7fd5d2c30640 1 -- 172.21.15.117:0/3658509248 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5cc135980 con 0x7fd5cc0b07e0 2024-05-15T13:20:29.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.031+0000 7fd5c97fa640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:20:29.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.031+0000 7fd5c97fa640 1 -- 172.21.15.117:0/3658509248 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5c4039040 con 0x7fd5cc0b07e0 2024-05-15T13:20:29.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.031+0000 7fd5c97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:29.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.031+0000 7fd5c97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T13:20:29.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.031+0000 7fd5c97fa640 10 monclient: dump: 2024-05-15T13:20:29.044 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:29.044 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:29.044 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:29.044 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:29.044 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:29.044 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:29.044 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:29.044 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:29.044 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:29.045 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:29.045 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.031+0000 7fd5caffd640 4 set_mon_vals no callback set 2024-05-15T13:20:29.045 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.031+0000 7fd5d2c30640 10 client.22147.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:20:29.045 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.031+0000 7fd5d2c30640 10 monclient: _renew_subs 2024-05-15T13:20:29.045 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.031+0000 7fd5d2c30640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T13:20:29.045 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.031+0000 7fd5d2c30640 1 -- 172.21.15.117:0/3658509248 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5cc1d8170 con 0x7fd5cc0b07e0 2024-05-15T13:20:29.045 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.031+0000 7fd5d2c30640 1 librados: init done 2024-05-15T13:20:29.045 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.032+0000 7fd5d2c30640 10 client.22147.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:20:29.045 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.032+0000 7fd5d2c30640 15 client.22147.objecter _session_command_op_assign -1 1 2024-05-15T13:20:29.045 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.032+0000 7fd5d2c30640 20 client.22147.objecter _get_session osd=-1 returning homeless 2024-05-15T13:20:29.045 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.032+0000 7fd5d2c30640 10 client.22147.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:20:29.045 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.032+0000 7fd5d2c30640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T13:20:29.046 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.032+0000 7fd5d2c30640 1 -- 172.21.15.117:0/3658509248 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd5cc04f0e0 con 0x7fd5cc0b07e0 2024-05-15T13:20:29.048 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.049+0000 7fd5c97fa640 1 -- 172.21.15.117:0/3658509248 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fd5c4047020 con 0x7fd5cc0b07e0 2024-05-15T13:20:29.048 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.050+0000 7fd5c97fa640 1 --2- 172.21.15.117:0/3658509248 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fd5b006ce90 0x7fd5b006f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:29.048 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.050+0000 7fd5d09cc640 1 --2- 172.21.15.117:0/3658509248 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fd5b006ce90 0x7fd5b006f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:29.048 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.050+0000 7fd5d09cc640 10 monclient: get_auth_request con 0x7fd5b006ce90 auth_method 0 2024-05-15T13:20:29.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.050+0000 7fd5c97fa640 10 client.22147.objecter ms_handle_connect 0x7fd5b006ce90 2024-05-15T13:20:29.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.050+0000 7fd5d09cc640 1 --2- 172.21.15.117:0/3658509248 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fd5b006ce90 0x7fd5b006f350 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fd5c0004e40 tx=0x7fd5c0007f10 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:29.053 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.053+0000 7fd5c97fa640 1 -- 172.21.15.117:0/3658509248 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(1225..1225 src has 912..1225) v4 ==== 6365+0+0 (secure 0 0 0) 0x7fd5c40798f0 con 0x7fd5cc0b07e0 2024-05-15T13:20:29.053 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.053+0000 7fd5c97fa640 10 client.22147.objecter ms_dispatch 0x7fd5cc000b90 osd_map(1225..1225 src has 912..1225) v4 2024-05-15T13:20:29.053 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.053+0000 7fd5c97fa640 3 client.22147.objecter handle_osd_map got epochs [1225,1225] > 0 2024-05-15T13:20:29.053 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.053+0000 7fd5c97fa640 3 client.22147.objecter handle_osd_map decoding full epoch 1225 2024-05-15T13:20:29.053 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.054+0000 7fd5c97fa640 10 client.22147.objecter checking command 1 2024-05-15T13:20:29.053 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.054+0000 7fd5c97fa640 1 --2- 172.21.15.117:0/3658509248 >> [v2:172.21.15.117:6818/3864975175,v1:172.21.15.117:6819/3864975175] conn(0x7fd5b0074c00 0x7fd5b00770a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:29.053 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.054+0000 7fd5c97fa640 20 client.22147.objecter _get_session s=0x7fd5b0074530 osd=1 3 2024-05-15T13:20:29.053 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.054+0000 7fd5c97fa640 20 client.22147.objecter put_session s=0x7fd5b0074530 osd=1 3 2024-05-15T13:20:29.053 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.054+0000 7fd5c97fa640 15 client.22147.objecter _session_command_op_remove -1 1 2024-05-15T13:20:29.053 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.054+0000 7fd5c97fa640 20 client.22147.objecter _get_session s=0x7fd5b0074530 osd=1 3 2024-05-15T13:20:29.053 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.054+0000 7fd5c97fa640 20 client.22147.objecter get_session s=0x7fd5b0074530 osd=1 3 2024-05-15T13:20:29.054 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.054+0000 7fd5c97fa640 15 client.22147.objecter _session_command_op_assign 1 1 2024-05-15T13:20:29.054 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.054+0000 7fd5c97fa640 20 client.22147.objecter put_session s=0x7fd5b0074530 osd=1 4 2024-05-15T13:20:29.054 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.054+0000 7fd5c97fa640 10 client.22147.objecter _send_command 1 2024-05-15T13:20:29.054 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.054+0000 7fd5c97fa640 1 -- 172.21.15.117:0/3658509248 --> [v2:172.21.15.117:6818/3864975175,v1:172.21.15.117:6819/3864975175] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd5c4042e50 con 0x7fd5b0074c00 2024-05-15T13:20:29.054 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.054+0000 7fd5c97fa640 20 client.22147.objecter dump_active .. 0 homeless 2024-05-15T13:20:29.054 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.054+0000 7fd5c97fa640 1 -- 172.21.15.117:0/3658509248 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_get_version_reply(handle=1 version=1225) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd5c4077af0 con 0x7fd5cc0b07e0 2024-05-15T13:20:29.054 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.054+0000 7fd5c97fa640 10 monclient: handle_get_version_reply finishing 1 version 1225 2024-05-15T13:20:29.054 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.054+0000 7fd5cbfff640 1 --2- 172.21.15.117:0/3658509248 >> [v2:172.21.15.117:6818/3864975175,v1:172.21.15.117:6819/3864975175] conn(0x7fd5b0074c00 0x7fd5b00770a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:29.054 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.055+0000 7fd5cbfff640 10 monclient: get_auth_request con 0x7fd5b0074c00 auth_method 0 2024-05-15T13:20:29.056 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.057+0000 7fd5cbfff640 1 --2- 172.21.15.117:0/3658509248 >> [v2:172.21.15.117:6818/3864975175,v1:172.21.15.117:6819/3864975175] conn(0x7fd5b0074c00 0x7fd5b00770a0 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:29.056 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.057+0000 7fd5c97fa640 10 client.22147.objecter ms_handle_connect 0x7fd5b0074c00 2024-05-15T13:20:29.079 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.081+0000 7fd5c97fa640 1 -- 172.21.15.117:0/3658509248 <== osd.1 v2:172.21.15.117:6818/3864975175 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd5c4042e50 con 0x7fd5b0074c00 2024-05-15T13:20:29.080 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.081+0000 7fd5c97fa640 10 client.22147.objecter ms_dispatch 0x7fd5cc000b90 command_reply(tid 1: 0 ) v1 2024-05-15T13:20:29.080 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.081+0000 7fd5c97fa640 10 client.22147.objecter _finish_command 1 = system:0 2024-05-15T13:20:29.080 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.081+0000 7fd5c97fa640 20 client.22147.objecter put_session s=0x7fd5b0074530 osd=1 4 2024-05-15T13:20:29.080 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.081+0000 7fd5c97fa640 15 client.22147.objecter _session_command_op_remove 1 1 2024-05-15T13:20:29.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.094+0000 7fd5d2c30640 10 client.22147.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-05-15T13:20:29.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.094+0000 7fd5d2c30640 15 client.22147.objecter _session_command_op_assign -1 2 2024-05-15T13:20:29.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.094+0000 7fd5d2c30640 20 client.22147.objecter _get_session s=0x7fd5b0074530 osd=1 3 2024-05-15T13:20:29.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.094+0000 7fd5d2c30640 20 client.22147.objecter put_session s=0x7fd5b0074530 osd=1 3 2024-05-15T13:20:29.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.094+0000 7fd5d2c30640 20 client.22147.objecter _get_session s=0x7fd5b0074530 osd=1 3 2024-05-15T13:20:29.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.094+0000 7fd5d2c30640 15 client.22147.objecter _session_command_op_remove -1 2 2024-05-15T13:20:29.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.094+0000 7fd5d2c30640 20 client.22147.objecter get_session s=0x7fd5b0074530 osd=1 3 2024-05-15T13:20:29.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.094+0000 7fd5d2c30640 15 client.22147.objecter _session_command_op_assign 1 2 2024-05-15T13:20:29.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.094+0000 7fd5d2c30640 20 client.22147.objecter put_session s=0x7fd5b0074530 osd=1 4 2024-05-15T13:20:29.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.094+0000 7fd5d2c30640 10 client.22147.objecter _send_command 2 2024-05-15T13:20:29.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.094+0000 7fd5d2c30640 1 -- 172.21.15.117:0/3658509248 --> [v2:172.21.15.117:6818/3864975175,v1:172.21.15.117:6819/3864975175] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fd5cc1d83a0 con 0x7fd5b0074c00 2024-05-15T13:20:29.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.109+0000 7fd5c97fa640 1 -- 172.21.15.117:0/3658509248 <== osd.1 v2:172.21.15.117:6818/3864975175 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fd5cc1d83a0 con 0x7fd5b0074c00 2024-05-15T13:20:29.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.109+0000 7fd5c97fa640 10 client.22147.objecter ms_dispatch 0x7fd5cc000b90 command_reply(tid 2: 0 ) v1 2024-05-15T13:20:29.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.109+0000 7fd5c97fa640 10 client.22147.objecter _finish_command 2 = system:0 2024-05-15T13:20:29.108 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.109+0000 7fd5c97fa640 20 client.22147.objecter put_session s=0x7fd5b0074530 osd=1 4 2024-05-15T13:20:29.108 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.109+0000 7fd5c97fa640 15 client.22147.objecter _session_command_op_remove 1 2 2024-05-15T13:20:29.108 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-05-15T13:20:29.108 INFO:teuthology.orchestra.run.smithi117.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-15T13:20:29.108 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-05-15T13:20:29.109 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.109+0000 7fd5d2c30640 10 librados: watch_flush enter 2024-05-15T13:20:29.110 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.109+0000 7fd5d2c30640 10 librados: watch_flush exit 2024-05-15T13:20:29.110 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.109+0000 7fd5d2c30640 10 client.22147.objecter close_session for osd.1 2024-05-15T13:20:29.110 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.109+0000 7fd5d2c30640 1 -- 172.21.15.117:0/3658509248 >> [v2:172.21.15.117:6818/3864975175,v1:172.21.15.117:6819/3864975175] conn(0x7fd5b0074c00 msgr2=0x7fd5b00770a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:29.110 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.109+0000 7fd5d2c30640 1 --2- 172.21.15.117:0/3658509248 >> [v2:172.21.15.117:6818/3864975175,v1:172.21.15.117:6819/3864975175] conn(0x7fd5b0074c00 0x7fd5b00770a0 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:29.110 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.109+0000 7fd5d2c30640 20 client.22147.objecter put_session s=0x7fd5b0074530 osd=1 1 2024-05-15T13:20:29.110 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.109+0000 7fd5d2c30640 20 client.22147.objecter shutdown clearing up homeless session... 2024-05-15T13:20:29.110 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.109+0000 7fd5d2c30640 10 client.22147.objecter successfully canceled tick 2024-05-15T13:20:29.110 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.109+0000 7fd5d2c30640 1 -- 172.21.15.117:0/3658509248 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fd5b006ce90 msgr2=0x7fd5b006f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:29.110 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.109+0000 7fd5d2c30640 1 --2- 172.21.15.117:0/3658509248 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fd5b006ce90 0x7fd5b006f350 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fd5c0004e40 tx=0x7fd5c0007f10 comp rx=0 tx=0).stop 2024-05-15T13:20:29.110 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.109+0000 7fd5d2c30640 10 monclient: shutdown 2024-05-15T13:20:29.110 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.109+0000 7fd5d2c30640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:20:29.110 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.109+0000 7fd5d2c30640 1 -- 172.21.15.117:0/3658509248 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd5cc0b07e0 msgr2=0x7fd5cc1d7c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:29.110 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.109+0000 7fd5d2c30640 1 --2- 172.21.15.117:0/3658509248 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd5cc0b07e0 0x7fd5cc1d7c30 secure :-1 s=READY pgs=5443 cs=0 l=1 rev1=1 crypto rx=0x7fd5c4033000 tx=0x7fd5c4004a40 comp rx=0 tx=0).stop 2024-05-15T13:20:29.111 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.109+0000 7fd5d11cd640 1 -- 172.21.15.117:0/3658509248 reap_dead start 2024-05-15T13:20:29.111 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.109+0000 7fd5d2c30640 1 -- 172.21.15.117:0/3658509248 shutdown_connections 2024-05-15T13:20:29.111 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.109+0000 7fd5d2c30640 1 -- 172.21.15.117:0/3658509248 >> 172.21.15.117:0/3658509248 conn(0x7fd5cc0b2ac0 msgr2=0x7fd5cc0b0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:20:29.111 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.109+0000 7fd5d2c30640 1 -- 172.21.15.117:0/3658509248 shutdown_connections 2024-05-15T13:20:29.111 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.109+0000 7fd5d2c30640 1 -- 172.21.15.117:0/3658509248 wait complete. 2024-05-15T13:20:29.111 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.109+0000 7fd5d2c30640 1 librados: shutdown 2024-05-15T13:20:29.119 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-05-15T13:20:29.217 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.219+0000 7f1a483fb640 10 monclient: get_monmap_and_config 2024-05-15T13:20:29.218 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.219+0000 7f1a483fb640 10 monclient: build_initial_monmap 2024-05-15T13:20:29.218 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.219+0000 7f1a483fb640 10 monclient: monmap: 2024-05-15T13:20:29.218 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T13:20:29.218 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:29.218 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T13:20:29.220301+0000 2024-05-15T13:20:29.218 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T13:20:29.220301+0000 2024-05-15T13:20:29.218 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T13:20:29.218 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:29.218 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T13:20:29.218 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T13:20:29.218 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T13:20:29.218 INFO:teuthology.orchestra.run.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T13:20:29.218 INFO:teuthology.orchestra.run.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T13:20:29.218 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:29.220 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.219+0000 7f1a483fb640 1 Processor -- start 2024-05-15T13:20:29.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.219+0000 7f1a483fb640 1 -- start start 2024-05-15T13:20:29.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.219+0000 7f1a483fb640 10 monclient: init 2024-05-15T13:20:29.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.220+0000 7f1a483fb640 10 monclient: _reopen_session rank -1 2024-05-15T13:20:29.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.220+0000 7f1a483fb640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-05-15T13:20:29.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.220+0000 7f1a483fb640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1a400b86e0 0x7f1a400aa0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:29.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.220+0000 7f1a483fb640 10 monclient(hunting): picked mon.noname-b con 0x7f1a400b86e0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T13:20:29.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.220+0000 7f1a483fb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1a400b1dd0 addr v1:172.21.15.178:6789/0 2024-05-15T13:20:29.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.220+0000 7f1a483fb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1a4013f870 addr v1:172.21.15.117:6789/0 2024-05-15T13:20:29.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.220+0000 7f1a483fb640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a400a9bc0 con 0x7f1a4013f870 2024-05-15T13:20:29.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.220+0000 7f1a483fb640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a400a9850 con 0x7f1a400b1dd0 2024-05-15T13:20:29.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.220+0000 7f1a483fb640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:29.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.220+0000 7f1a483fb640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a400b8ac0 con 0x7f1a400b86e0 2024-05-15T13:20:29.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.220+0000 7f1a483fb640 10 monclient(hunting): _renew_subs 2024-05-15T13:20:29.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.220+0000 7f1a483fb640 10 monclient(hunting): authenticate will time out at 3592.462484s 2024-05-15T13:20:29.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.221+0000 7f1a46197640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1a400b86e0 0x7f1a400aa0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:29.222 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.221+0000 7f1a45996640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f1a400b1dd0 0x7f1a40145bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.117:37330/0 (socket says 172.21.15.117:37330) 2024-05-15T13:20:29.222 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.221+0000 7f1a45996640 1 -- 172.21.15.117:0/556047630 learned_addr learned my addr 172.21.15.117:0/556047630 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-05-15T13:20:29.222 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.221+0000 7f1a46197640 10 monclient(hunting): get_auth_request con 0x7f1a400b86e0 auth_method 0 2024-05-15T13:20:29.222 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.221+0000 7f1a46197640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:29.222 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.221+0000 7f1a46197640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:29.222 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.221+0000 7f1a46197640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:29.222 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.222+0000 7f1a46197640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:29.222 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.222+0000 7f1a46197640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:29.222 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.222+0000 7f1a46197640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:29.223 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.223+0000 7f1a45195640 1 -- 172.21.15.117:0/556047630 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2873953686 0 0) 0x7f1a400a9850 con 0x7f1a400b1dd0 2024-05-15T13:20:29.223 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.223+0000 7f1a45195640 10 monclient(hunting): _negotiate 2024-05-15T13:20:29.223 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.223+0000 7f1a45195640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:29.223 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.223+0000 7f1a45195640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:29.223 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.223+0000 7f1a45195640 10 monclient(hunting): my global_id is 22150 2024-05-15T13:20:29.223 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.223+0000 7f1a45195640 1 -- 172.21.15.117:0/556047630 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a28003520 con 0x7f1a400b1dd0 2024-05-15T13:20:29.223 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.223+0000 7f1a46197640 10 monclient(hunting): handle_auth_done global_id 18734 payload 274 2024-05-15T13:20:29.223 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.223+0000 7f1a46197640 1 -- 172.21.15.117:0/556047630 >> v1:172.21.15.178:6789/0 conn(0x7f1a400b1dd0 legacy=0x7f1a40145bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:29.223 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.223+0000 7f1a46197640 1 -- 172.21.15.117:0/556047630 >> v1:172.21.15.117:6789/0 conn(0x7f1a4013f870 legacy=0x7f1a4013fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:29.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.223+0000 7f1a46197640 10 monclient: _finish_hunting 0 2024-05-15T13:20:29.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.223+0000 7f1a46197640 1 monclient: found mon.noname-b 2024-05-15T13:20:29.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.223+0000 7f1a46197640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:20:29.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.223+0000 7f1a46197640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.117:3301/0 2024-05-15T13:20:29.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.223+0000 7f1a46197640 1 -- 172.21.15.117:0/556047630 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a400b3a60 con 0x7f1a400b86e0 2024-05-15T13:20:29.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.225+0000 7f1a46197640 1 --2- 172.21.15.117:0/556047630 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1a400b86e0 0x7f1a400aa0e0 secure :-1 s=READY pgs=3382 cs=0 l=1 rev1=1 crypto rx=0x7f1a380097d0 tx=0x7f1a3802b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9263aa6b5138672 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:29.228 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.229+0000 7f1a45195640 1 -- 172.21.15.117:0/556047630 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a38035020 con 0x7f1a400b86e0 2024-05-15T13:20:29.228 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.229+0000 7f1a45195640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:29.228 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.229+0000 7f1a45195640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T13:20:29.228 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.229+0000 7f1a45195640 10 monclient: dump: 2024-05-15T13:20:29.228 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:29.228 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:29.229 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:29.229 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:29.229 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:29.229 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:29.229 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:29.229 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:29.229 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:29.229 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:29.229 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.229+0000 7f1a45195640 10 monclient: _finish_auth 0 2024-05-15T13:20:29.229 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.229+0000 7f1a45195640 10 monclient: _check_auth_tickets 2024-05-15T13:20:29.229 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.229+0000 7f1a45195640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:20:29.229 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.229+0000 7f1a45195640 1 -- 172.21.15.117:0/556047630 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a38031800 con 0x7f1a400b86e0 2024-05-15T13:20:29.229 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.229+0000 7f1a483fb640 5 monclient: authenticate success, global_id 18734 2024-05-15T13:20:29.229 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.229+0000 7f1a483fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T13:20:29.229 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.229+0000 7f1a45195640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:20:29.229 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.229+0000 7f1a483fb640 10 monclient: get_monmap_and_config success 2024-05-15T13:20:29.230 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.229+0000 7f1a483fb640 4 set_mon_vals no callback set 2024-05-15T13:20:29.230 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.229+0000 7f1a45195640 1 -- 172.21.15.117:0/556047630 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a38031bd0 con 0x7f1a400b86e0 2024-05-15T13:20:29.230 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.229+0000 7f1a45195640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:29.230 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.229+0000 7f1a45195640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T13:20:29.230 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.229+0000 7f1a45195640 10 monclient: dump: 2024-05-15T13:20:29.230 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:29.230 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:29.230 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:29.230 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:29.230 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:29.230 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:29.230 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:29.230 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:29.230 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:29.231 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:29.231 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.229+0000 7f1a483fb640 10 monclient: shutdown 2024-05-15T13:20:29.231 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.229+0000 7f1a483fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:20:29.231 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.229+0000 7f1a483fb640 1 -- 172.21.15.117:0/556047630 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1a400b86e0 msgr2=0x7f1a400aa0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:29.231 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.229+0000 7f1a483fb640 1 --2- 172.21.15.117:0/556047630 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1a400b86e0 0x7f1a400aa0e0 secure :-1 s=READY pgs=3382 cs=0 l=1 rev1=1 crypto rx=0x7f1a380097d0 tx=0x7f1a3802b8a0 comp rx=0 tx=0).stop 2024-05-15T13:20:29.231 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.229+0000 7f1a483fb640 1 -- 172.21.15.117:0/556047630 shutdown_connections 2024-05-15T13:20:29.231 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.229+0000 7f1a483fb640 1 --2- 172.21.15.117:0/556047630 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1a400b86e0 0x7f1a400aa0e0 unknown :-1 s=CLOSED pgs=3382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:29.231 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.230+0000 7f1a483fb640 1 -- 172.21.15.117:0/556047630 >> 172.21.15.117:0/556047630 conn(0x7f1a40066f00 msgr2=0x7f1a40067310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:20:29.231 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.230+0000 7f1a483fb640 1 -- 172.21.15.117:0/556047630 shutdown_connections 2024-05-15T13:20:29.231 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.230+0000 7f1a483fb640 1 -- 172.21.15.117:0/556047630 wait complete. 2024-05-15T13:20:29.234 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.230+0000 7f1a483fb640 10 monclient: build_initial_monmap 2024-05-15T13:20:29.234 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.230+0000 7f1a483fb640 10 monclient: monmap: 2024-05-15T13:20:29.234 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T13:20:29.234 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T13:20:29.234 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-05-15T13:20:29.234 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-05-15T13:20:29.234 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T13:20:29.234 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:29.234 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T13:20:29.234 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T13:20:29.234 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T13:20:29.234 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:29.235 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.231+0000 7f1a483fb640 1 librados: starting msgr at 2024-05-15T13:20:29.235 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.231+0000 7f1a483fb640 1 librados: starting objecter 2024-05-15T13:20:29.235 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.231+0000 7f1a483fb640 1 Processor -- start 2024-05-15T13:20:29.235 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.231+0000 7f1a483fb640 1 -- start start 2024-05-15T13:20:29.235 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.231+0000 7f1a483fb640 1 librados: setting wanted keys 2024-05-15T13:20:29.235 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.231+0000 7f1a483fb640 1 librados: calling monclient init 2024-05-15T13:20:29.235 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.231+0000 7f1a483fb640 10 monclient: init 2024-05-15T13:20:29.235 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.231+0000 7f1a483fb640 10 monclient: _reopen_session rank -1 2024-05-15T13:20:29.235 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.231+0000 7f1a483fb640 10 monclient: _add_conns ranks=[1,0,2] 2024-05-15T13:20:29.235 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.231+0000 7f1a483fb640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1a400b1dd0 0x7f1a400b6e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:29.235 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.231+0000 7f1a483fb640 10 monclient(hunting): picked mon.noname-b con 0x7f1a400b1dd0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T13:20:29.235 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.231+0000 7f1a483fb640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a400b86e0 0x7f1a40142ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:29.235 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.231+0000 7f1a483fb640 10 monclient(hunting): picked mon.noname-a con 0x7f1a400b86e0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T13:20:29.235 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.231+0000 7f1a483fb640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1a4013f870 0x7f1a40141d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:29.235 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.232+0000 7f1a483fb640 10 monclient(hunting): picked mon.noname-c con 0x7f1a4013f870 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T13:20:29.236 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.232+0000 7f1a483fb640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:29.236 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.232+0000 7f1a483fb640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a4014e130 con 0x7f1a400b86e0 2024-05-15T13:20:29.236 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.232+0000 7f1a483fb640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:29.236 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.232+0000 7f1a483fb640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a400aa620 con 0x7f1a400b1dd0 2024-05-15T13:20:29.236 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.232+0000 7f1a483fb640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:29.236 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.232+0000 7f1a483fb640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a4014e2b0 con 0x7f1a4013f870 2024-05-15T13:20:29.236 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.232+0000 7f1a483fb640 10 monclient(hunting): _renew_subs 2024-05-15T13:20:29.236 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.232+0000 7f1a483fb640 10 monclient(hunting): authenticate will time out at 3592.473816s 2024-05-15T13:20:29.236 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.232+0000 7f1a46998640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1a4013f870 0x7f1a40141d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:29.236 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.232+0000 7f1a45996640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a400b86e0 0x7f1a40142ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:29.236 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.232+0000 7f1a46197640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1a400b1dd0 0x7f1a400b6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:29.236 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.232+0000 7f1a45996640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a400b86e0 0x7f1a40142ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:54812/0 (socket says 172.21.15.117:54812) 2024-05-15T13:20:29.236 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.232+0000 7f1a45996640 1 -- 172.21.15.117:0/1764129313 learned_addr learned my addr 172.21.15.117:0/1764129313 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T13:20:29.236 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.232+0000 7f1a45996640 10 monclient(hunting): get_auth_request con 0x7f1a400b86e0 auth_method 0 2024-05-15T13:20:29.236 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.232+0000 7f1a45996640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:29.237 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.232+0000 7f1a45996640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:29.237 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.232+0000 7f1a45996640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:29.237 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.232+0000 7f1a46998640 10 monclient(hunting): get_auth_request con 0x7f1a4013f870 auth_method 0 2024-05-15T13:20:29.237 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.232+0000 7f1a46998640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:29.237 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.232+0000 7f1a46998640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:29.237 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.232+0000 7f1a46998640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:29.237 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.232+0000 7f1a46197640 10 monclient(hunting): get_auth_request con 0x7f1a400b1dd0 auth_method 0 2024-05-15T13:20:29.237 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.232+0000 7f1a46197640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:29.237 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.232+0000 7f1a46197640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:29.237 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.232+0000 7f1a46197640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:29.237 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.233+0000 7f1a46197640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:29.237 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.233+0000 7f1a46197640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:29.237 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.233+0000 7f1a46197640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:29.237 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.233+0000 7f1a45996640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:29.237 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.233+0000 7f1a45996640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:29.238 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.233+0000 7f1a45996640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:29.238 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.233+0000 7f1a46998640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:29.238 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.233+0000 7f1a46998640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:29.238 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.233+0000 7f1a46998640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:29.239 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.236+0000 7f1a46998640 10 monclient(hunting): handle_auth_done global_id 18737 payload 824 2024-05-15T13:20:29.239 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.236+0000 7f1a46998640 1 -- 172.21.15.117:0/1764129313 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1a400b1dd0 msgr2=0x7f1a400b6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:29.239 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.236+0000 7f1a46998640 1 --2- 172.21.15.117:0/1764129313 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1a400b1dd0 0x7f1a400b6e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:29.239 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.236+0000 7f1a46998640 1 -- 172.21.15.117:0/1764129313 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a400b86e0 msgr2=0x7f1a40142ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:29.239 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.236+0000 7f1a46998640 1 --2- 172.21.15.117:0/1764129313 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a400b86e0 0x7f1a40142ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:29.239 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.236+0000 7f1a46998640 10 monclient: _finish_hunting 0 2024-05-15T13:20:29.239 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.236+0000 7f1a46998640 1 monclient: found mon.noname-c 2024-05-15T13:20:29.239 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.236+0000 7f1a46998640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:20:29.239 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.236+0000 7f1a46998640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.117:3301/0 2024-05-15T13:20:29.239 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.236+0000 7f1a46998640 1 -- 172.21.15.117:0/1764129313 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a401425c0 con 0x7f1a4013f870 2024-05-15T13:20:29.239 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.238+0000 7f1a46998640 1 --2- 172.21.15.117:0/1764129313 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1a4013f870 0x7f1a40141d80 secure :-1 s=READY pgs=3383 cs=0 l=1 rev1=1 crypto rx=0x7f1a3000c700 tx=0x7f1a3000cbc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:29.239 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.238+0000 7f1a3effd640 10 client.?.objecter ms_handle_connect 0x7f1a4013f870 2024-05-15T13:20:29.239 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.238+0000 7f1a3effd640 10 client.?.objecter resend_mon_ops 2024-05-15T13:20:29.240 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.241+0000 7f1a3effd640 1 -- 172.21.15.117:0/1764129313 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a30003b90 con 0x7f1a4013f870 2024-05-15T13:20:29.240 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.241+0000 7f1a3effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:29.240 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.241+0000 7f1a3effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T13:20:29.241 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.241+0000 7f1a3effd640 10 monclient: dump: 2024-05-15T13:20:29.241 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:29.241 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:29.241 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:29.241 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:29.241 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:29.241 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:29.241 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:29.241 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:29.241 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:29.241 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:29.241 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.241+0000 7f1a3effd640 10 monclient: _finish_auth 0 2024-05-15T13:20:29.242 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.241+0000 7f1a3effd640 10 monclient: _check_auth_tickets 2024-05-15T13:20:29.242 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.241+0000 7f1a3effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:20:29.242 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.241+0000 7f1a3effd640 1 -- 172.21.15.117:0/1764129313 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a30003d50 con 0x7f1a4013f870 2024-05-15T13:20:29.242 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.241+0000 7f1a3effd640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:20:29.242 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.241+0000 7f1a3effd640 1 -- 172.21.15.117:0/1764129313 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a30005310 con 0x7f1a4013f870 2024-05-15T13:20:29.242 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.241+0000 7f1a3effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:29.242 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.241+0000 7f1a3effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T13:20:29.242 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.241+0000 7f1a3effd640 10 monclient: dump: 2024-05-15T13:20:29.242 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:29.242 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:29.242 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:29.242 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:29.242 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:29.242 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:29.243 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:29.243 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:29.243 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:29.243 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:29.243 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.241+0000 7f1a483fb640 5 monclient: authenticate success, global_id 18737 2024-05-15T13:20:29.243 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.241+0000 7f1a483fb640 10 monclient: _renew_subs 2024-05-15T13:20:29.243 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.241+0000 7f1a483fb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T13:20:29.243 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.241+0000 7f1a483fb640 1 -- 172.21.15.117:0/1764129313 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a401422c0 con 0x7f1a4013f870 2024-05-15T13:20:29.243 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.241+0000 7f1a44994640 4 set_mon_vals no callback set 2024-05-15T13:20:29.243 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.241+0000 7f1a483fb640 10 client.18737.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:20:29.243 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.241+0000 7f1a483fb640 10 monclient: _renew_subs 2024-05-15T13:20:29.243 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.241+0000 7f1a483fb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T13:20:29.243 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.241+0000 7f1a483fb640 1 -- 172.21.15.117:0/1764129313 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a40135b50 con 0x7f1a4013f870 2024-05-15T13:20:29.243 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.241+0000 7f1a483fb640 1 librados: init done 2024-05-15T13:20:29.244 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.243+0000 7f1a483fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T13:20:29.244 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.243+0000 7f1a483fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:20:29.244 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.243+0000 7f1a483fb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T13:20:29.244 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.243+0000 7f1a483fb640 1 -- 172.21.15.117:0/1764129313 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a4004f0e0 con 0x7f1a4013f870 2024-05-15T13:20:29.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.494+0000 7f1a3effd640 1 -- 172.21.15.117:0/1764129313 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f1a300054b0 con 0x7f1a4013f870 2024-05-15T13:20:29.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.494+0000 7f1a3effd640 1 --2- 172.21.15.117:0/1764129313 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f1a2c06ce90 0x7f1a2c06f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:29.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.494+0000 7f1a46197640 1 --2- 172.21.15.117:0/1764129313 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f1a2c06ce90 0x7f1a2c06f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:29.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.494+0000 7f1a46197640 10 monclient: get_auth_request con 0x7f1a2c06ce90 auth_method 0 2024-05-15T13:20:29.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.495+0000 7f1a3effd640 10 client.18737.objecter ms_handle_connect 0x7f1a2c06ce90 2024-05-15T13:20:29.494 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.495+0000 7f1a46197640 1 --2- 172.21.15.117:0/1764129313 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f1a2c06ce90 0x7f1a2c06f350 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f1a38036000 tx=0x7f1a38008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:29.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.508+0000 7f1a3effd640 1 -- 172.21.15.117:0/1764129313 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(1225..1225 src has 912..1225) v4 ==== 6365+0+0 (secure 0 0 0) 0x7f1a300057a0 con 0x7f1a4013f870 2024-05-15T13:20:29.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.508+0000 7f1a3effd640 10 client.18737.objecter ms_dispatch 0x7f1a40000b90 osd_map(1225..1225 src has 912..1225) v4 2024-05-15T13:20:29.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.508+0000 7f1a3effd640 3 client.18737.objecter handle_osd_map got epochs [1225,1225] > 0 2024-05-15T13:20:29.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.508+0000 7f1a3effd640 3 client.18737.objecter handle_osd_map decoding full epoch 1225 2024-05-15T13:20:29.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.508+0000 7f1a3effd640 20 client.18737.objecter dump_active .. 0 homeless 2024-05-15T13:20:29.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.508+0000 7f1a3effd640 1 -- 172.21.15.117:0/1764129313 <== mon.2 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f1a30059210 con 0x7f1a4013f870 2024-05-15T13:20:29.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.508+0000 7f1a3effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:20:29.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.508+0000 7f1a3effd640 10 monclient: _finish_command 1 = system:0 2024-05-15T13:20:29.679 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.680+0000 7f1a483fb640 1 -- 172.21.15.117:0/1764129313 --> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f1a40142790 con 0x7f1a2c06ce90 2024-05-15T13:20:29.679 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.681+0000 7f1a3effd640 1 -- 172.21.15.117:0/1764129313 <== mgr.4107 v2:172.21.15.117:6800/1785766523 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f1a40142790 con 0x7f1a2c06ce90 2024-05-15T13:20:29.680 INFO:teuthology.orchestra.run.smithi117.stderr:instructed osd(s) 1 to deep-scrub 2024-05-15T13:20:29.682 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.683+0000 7f1a483fb640 10 librados: watch_flush enter 2024-05-15T13:20:29.682 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.683+0000 7f1a483fb640 10 librados: watch_flush exit 2024-05-15T13:20:29.682 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.683+0000 7f1a483fb640 20 client.18737.objecter shutdown clearing up homeless session... 2024-05-15T13:20:29.682 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.683+0000 7f1a483fb640 10 client.18737.objecter successfully canceled tick 2024-05-15T13:20:29.682 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.683+0000 7f1a483fb640 1 -- 172.21.15.117:0/1764129313 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f1a2c06ce90 msgr2=0x7f1a2c06f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:29.682 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.683+0000 7f1a483fb640 1 --2- 172.21.15.117:0/1764129313 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f1a2c06ce90 0x7f1a2c06f350 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f1a38036000 tx=0x7f1a38008030 comp rx=0 tx=0).stop 2024-05-15T13:20:29.683 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.683+0000 7f1a483fb640 10 monclient: shutdown 2024-05-15T13:20:29.683 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.683+0000 7f1a483fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:20:29.683 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.683+0000 7f1a483fb640 1 -- 172.21.15.117:0/1764129313 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1a4013f870 msgr2=0x7f1a40141d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:29.683 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.683+0000 7f1a483fb640 1 --2- 172.21.15.117:0/1764129313 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1a4013f870 0x7f1a40141d80 secure :-1 s=READY pgs=3383 cs=0 l=1 rev1=1 crypto rx=0x7f1a3000c700 tx=0x7f1a3000cbc0 comp rx=0 tx=0).stop 2024-05-15T13:20:29.683 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.683+0000 7f1a483fb640 1 -- 172.21.15.117:0/1764129313 shutdown_connections 2024-05-15T13:20:29.683 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.683+0000 7f1a483fb640 1 --2- 172.21.15.117:0/1764129313 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1a4013f870 0x7f1a40141d80 unknown :-1 s=CLOSED pgs=3383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:29.683 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.683+0000 7f1a483fb640 1 --2- 172.21.15.117:0/1764129313 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a400b86e0 0x7f1a40142ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:29.683 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.683+0000 7f1a483fb640 1 --2- 172.21.15.117:0/1764129313 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f1a2c06ce90 0x7f1a2c06f350 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:29.683 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.683+0000 7f1a483fb640 1 --2- 172.21.15.117:0/1764129313 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1a400b1dd0 0x7f1a400b6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:29.683 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.683+0000 7f1a483fb640 1 -- 172.21.15.117:0/1764129313 >> 172.21.15.117:0/1764129313 conn(0x7f1a40066f00 msgr2=0x7f1a4013ca90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:20:29.683 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.683+0000 7f1a483fb640 1 -- 172.21.15.117:0/1764129313 shutdown_connections 2024-05-15T13:20:29.683 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.683+0000 7f1a483fb640 1 -- 172.21.15.117:0/1764129313 wait complete. 2024-05-15T13:20:29.683 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.683+0000 7f1a483fb640 1 librados: shutdown 2024-05-15T13:20:29.694 INFO:tasks.ceph:Scrubbing osd.2 2024-05-15T13:20:29.694 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-05-15T13:20:29.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.799+0000 7fd0efab7640 10 monclient: get_monmap_and_config 2024-05-15T13:20:29.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.799+0000 7fd0efab7640 10 monclient: build_initial_monmap 2024-05-15T13:20:29.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.799+0000 7fd0efab7640 10 monclient: monmap: 2024-05-15T13:20:29.798 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T13:20:29.799 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:29.799 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T13:20:29.800990+0000 2024-05-15T13:20:29.799 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T13:20:29.800990+0000 2024-05-15T13:20:29.799 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T13:20:29.799 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:29.799 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T13:20:29.799 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T13:20:29.799 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T13:20:29.799 INFO:teuthology.orchestra.run.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T13:20:29.799 INFO:teuthology.orchestra.run.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T13:20:29.799 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:29.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.800+0000 7fd0efab7640 1 Processor -- start 2024-05-15T13:20:29.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.800+0000 7fd0efab7640 1 -- start start 2024-05-15T13:20:29.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.800+0000 7fd0efab7640 10 monclient: init 2024-05-15T13:20:29.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.801+0000 7fd0efab7640 10 monclient: _reopen_session rank -1 2024-05-15T13:20:29.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.801+0000 7fd0efab7640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-05-15T13:20:29.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.801+0000 7fd0efab7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd0e80a9b60 addr v1:172.21.15.117:6789/0 2024-05-15T13:20:29.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.801+0000 7fd0efab7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd0e80aa3c0 addr v1:172.21.15.178:6789/0 2024-05-15T13:20:29.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.801+0000 7fd0efab7640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd0e8140110 0x7fd0e8143190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:29.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.801+0000 7fd0efab7640 10 monclient(hunting): picked mon.noname-b con 0x7fd0e8140110 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T13:20:29.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.801+0000 7fd0efab7640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0e80aa1a0 con 0x7fd0e80a9b60 2024-05-15T13:20:29.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.801+0000 7fd0efab7640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0e81436d0 con 0x7fd0e80aa3c0 2024-05-15T13:20:29.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.801+0000 7fd0efab7640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:29.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.801+0000 7fd0efab7640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e80a9f40 con 0x7fd0e8140110 2024-05-15T13:20:29.802 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.801+0000 7fd0efab7640 10 monclient(hunting): _renew_subs 2024-05-15T13:20:29.802 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.801+0000 7fd0efab7640 10 monclient(hunting): authenticate will time out at 3593.043238s 2024-05-15T13:20:29.802 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.801+0000 7fd0ed853640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fd0e80a9b60 0x7fd0e813fa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:41452/0 (socket says 172.21.15.117:41452) 2024-05-15T13:20:29.802 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.801+0000 7fd0ed853640 1 -- 172.21.15.117:0/560973419 learned_addr learned my addr 172.21.15.117:0/560973419 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-05-15T13:20:29.802 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.801+0000 7fd0ee054640 1 --2- 172.21.15.117:0/560973419 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd0e8140110 0x7fd0e8143190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:29.802 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.802+0000 7fd0ee054640 10 monclient(hunting): get_auth_request con 0x7fd0e8140110 auth_method 0 2024-05-15T13:20:29.802 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.802+0000 7fd0ee054640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:29.802 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.802+0000 7fd0ee054640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:29.802 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.802+0000 7fd0ee054640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:29.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.803+0000 7fd0ee054640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:29.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.803+0000 7fd0ee054640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:29.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.803+0000 7fd0ee054640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:29.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.804+0000 7fd0ec851640 1 -- 172.21.15.117:0/560973419 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3691018695 0 0) 0x7fd0e80aa1a0 con 0x7fd0e80a9b60 2024-05-15T13:20:29.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.804+0000 7fd0ec851640 10 monclient(hunting): _negotiate 2024-05-15T13:20:29.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.804+0000 7fd0ec851640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:29.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.804+0000 7fd0ec851640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:29.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.804+0000 7fd0ec851640 10 monclient(hunting): my global_id is 16029 2024-05-15T13:20:29.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.804+0000 7fd0ec851640 1 -- 172.21.15.117:0/560973419 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0d0003520 con 0x7fd0e80a9b60 2024-05-15T13:20:29.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.804+0000 7fd0ec851640 1 -- 172.21.15.117:0/560973419 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1046053070 0 0) 0x7fd0e81436d0 con 0x7fd0e80aa3c0 2024-05-15T13:20:29.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.804+0000 7fd0ec851640 10 monclient(hunting): _negotiate 2024-05-15T13:20:29.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.804+0000 7fd0ec851640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:29.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.804+0000 7fd0ec851640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:29.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.804+0000 7fd0ec851640 10 monclient(hunting): my global_id is 22156 2024-05-15T13:20:29.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.804+0000 7fd0ec851640 1 -- 172.21.15.117:0/560973419 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0e80aa1a0 con 0x7fd0e80aa3c0 2024-05-15T13:20:29.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.804+0000 7fd0ee054640 10 monclient(hunting): handle_auth_done global_id 18740 payload 274 2024-05-15T13:20:29.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.804+0000 7fd0ee054640 1 -- 172.21.15.117:0/560973419 >> v1:172.21.15.178:6789/0 conn(0x7fd0e80aa3c0 legacy=0x7fd0e8142a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:29.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.804+0000 7fd0ee054640 1 -- 172.21.15.117:0/560973419 >> v1:172.21.15.117:6789/0 conn(0x7fd0e80a9b60 legacy=0x7fd0e813fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:29.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.804+0000 7fd0ee054640 10 monclient: _finish_hunting 0 2024-05-15T13:20:29.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.804+0000 7fd0ee054640 1 monclient: found mon.noname-b 2024-05-15T13:20:29.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.804+0000 7fd0ee054640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:20:29.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.804+0000 7fd0ee054640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.117:3301/0 2024-05-15T13:20:29.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.804+0000 7fd0ee054640 1 -- 172.21.15.117:0/560973419 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0e80b25e0 con 0x7fd0e8140110 2024-05-15T13:20:29.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.806+0000 7fd0ee054640 1 --2- 172.21.15.117:0/560973419 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd0e8140110 0x7fd0e8143190 secure :-1 s=READY pgs=3384 cs=0 l=1 rev1=1 crypto rx=0x7fd0d80097d0 tx=0x7fd0d802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c175a1131d709f6c server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:29.809 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.810+0000 7fd0ec851640 1 -- 172.21.15.117:0/560973419 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d8035020 con 0x7fd0e8140110 2024-05-15T13:20:29.809 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.810+0000 7fd0ec851640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:29.809 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.810+0000 7fd0ec851640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T13:20:29.809 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.810+0000 7fd0ec851640 10 monclient: dump: 2024-05-15T13:20:29.809 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:29.809 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:29.809 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:29.809 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:29.809 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:29.810 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:29.810 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:29.810 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:29.810 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:29.810 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:29.810 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.810+0000 7fd0ec851640 10 monclient: _finish_auth 0 2024-05-15T13:20:29.810 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.810+0000 7fd0ec851640 10 monclient: _check_auth_tickets 2024-05-15T13:20:29.810 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.810+0000 7fd0ec851640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:20:29.810 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.810+0000 7fd0ec851640 1 -- 172.21.15.117:0/560973419 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0d8031890 con 0x7fd0e8140110 2024-05-15T13:20:29.810 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.810+0000 7fd0efab7640 5 monclient: authenticate success, global_id 18740 2024-05-15T13:20:29.810 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.810+0000 7fd0efab7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T13:20:29.810 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.810+0000 7fd0ec851640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:20:29.811 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.810+0000 7fd0efab7640 10 monclient: get_monmap_and_config success 2024-05-15T13:20:29.811 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.810+0000 7fd0efab7640 4 set_mon_vals no callback set 2024-05-15T13:20:29.811 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.810+0000 7fd0ec851640 1 -- 172.21.15.117:0/560973419 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d8031c60 con 0x7fd0e8140110 2024-05-15T13:20:29.811 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.810+0000 7fd0ec851640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:29.811 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.810+0000 7fd0ec851640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T13:20:29.811 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.810+0000 7fd0ec851640 10 monclient: dump: 2024-05-15T13:20:29.811 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:29.811 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:29.811 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:29.811 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:29.811 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:29.811 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:29.811 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:29.811 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:29.812 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:29.812 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:29.815 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.811+0000 7fd0efab7640 10 monclient: shutdown 2024-05-15T13:20:29.816 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.811+0000 7fd0efab7640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:20:29.816 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.811+0000 7fd0efab7640 1 -- 172.21.15.117:0/560973419 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd0e8140110 msgr2=0x7fd0e8143190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:29.816 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.811+0000 7fd0efab7640 1 --2- 172.21.15.117:0/560973419 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd0e8140110 0x7fd0e8143190 secure :-1 s=READY pgs=3384 cs=0 l=1 rev1=1 crypto rx=0x7fd0d80097d0 tx=0x7fd0d802b930 comp rx=0 tx=0).stop 2024-05-15T13:20:29.816 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.811+0000 7fd0efab7640 1 -- 172.21.15.117:0/560973419 shutdown_connections 2024-05-15T13:20:29.816 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.811+0000 7fd0efab7640 1 --2- 172.21.15.117:0/560973419 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd0e8140110 0x7fd0e8143190 unknown :-1 s=CLOSED pgs=3384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:29.816 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.811+0000 7fd0efab7640 1 -- 172.21.15.117:0/560973419 >> 172.21.15.117:0/560973419 conn(0x7fd0e80b7f90 msgr2=0x7fd0e80b83a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:20:29.816 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.811+0000 7fd0efab7640 1 -- 172.21.15.117:0/560973419 shutdown_connections 2024-05-15T13:20:29.816 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.811+0000 7fd0efab7640 1 -- 172.21.15.117:0/560973419 wait complete. 2024-05-15T13:20:29.816 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.812+0000 7fd0efab7640 10 monclient: build_initial_monmap 2024-05-15T13:20:29.816 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.812+0000 7fd0efab7640 10 monclient: monmap: 2024-05-15T13:20:29.816 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T13:20:29.816 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T13:20:29.816 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-05-15T13:20:29.817 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-05-15T13:20:29.817 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T13:20:29.817 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:29.817 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T13:20:29.817 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T13:20:29.817 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T13:20:29.817 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:29.818 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.812+0000 7fd0efab7640 1 librados: starting msgr at 2024-05-15T13:20:29.818 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.812+0000 7fd0efab7640 1 librados: starting objecter 2024-05-15T13:20:29.818 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.812+0000 7fd0efab7640 1 Processor -- start 2024-05-15T13:20:29.818 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.812+0000 7fd0efab7640 1 -- start start 2024-05-15T13:20:29.818 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.812+0000 7fd0efab7640 1 librados: setting wanted keys 2024-05-15T13:20:29.818 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.812+0000 7fd0efab7640 1 librados: calling monclient init 2024-05-15T13:20:29.818 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.812+0000 7fd0efab7640 10 monclient: init 2024-05-15T13:20:29.818 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.813+0000 7fd0efab7640 10 monclient: _reopen_session rank -1 2024-05-15T13:20:29.818 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.813+0000 7fd0efab7640 10 monclient: _add_conns ranks=[1,2,0] 2024-05-15T13:20:29.819 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.813+0000 7fd0efab7640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd0e80a9b60 0x7fd0e8146650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:29.819 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.813+0000 7fd0efab7640 10 monclient(hunting): picked mon.noname-b con 0x7fd0e80a9b60 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T13:20:29.819 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.813+0000 7fd0efab7640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd0e80aa3c0 0x7fd0e81352c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:29.819 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.813+0000 7fd0efab7640 10 monclient(hunting): picked mon.noname-c con 0x7fd0e80aa3c0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T13:20:29.819 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.813+0000 7fd0efab7640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd0e8140110 0x7fd0e8136e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:29.819 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.813+0000 7fd0efab7640 10 monclient(hunting): picked mon.noname-a con 0x7fd0e8140110 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T13:20:29.819 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.813+0000 7fd0efab7640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:29.820 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.813+0000 7fd0efab7640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e814e240 con 0x7fd0e8140110 2024-05-15T13:20:29.820 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.813+0000 7fd0efab7640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:29.820 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.813+0000 7fd0efab7640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e814e0c0 con 0x7fd0e80a9b60 2024-05-15T13:20:29.820 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.813+0000 7fd0efab7640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:29.820 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.813+0000 7fd0efab7640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e814e3c0 con 0x7fd0e80aa3c0 2024-05-15T13:20:29.820 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.813+0000 7fd0efab7640 10 monclient(hunting): _renew_subs 2024-05-15T13:20:29.820 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.813+0000 7fd0efab7640 10 monclient(hunting): authenticate will time out at 3593.055014s 2024-05-15T13:20:29.820 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.814+0000 7fd0ed853640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd0e80a9b60 0x7fd0e8146650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:29.820 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.814+0000 7fd0ed853640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd0e80a9b60 0x7fd0e8146650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:37290/0 (socket says 172.21.15.117:37290) 2024-05-15T13:20:29.820 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.814+0000 7fd0ed853640 1 -- 172.21.15.117:0/394009962 learned_addr learned my addr 172.21.15.117:0/394009962 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T13:20:29.820 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.814+0000 7fd0ed853640 10 monclient(hunting): get_auth_request con 0x7fd0e80a9b60 auth_method 0 2024-05-15T13:20:29.820 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.814+0000 7fd0ed853640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:29.820 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.814+0000 7fd0ed853640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:29.820 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.814+0000 7fd0ed853640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:29.820 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.814+0000 7fd0ee054640 1 --2- 172.21.15.117:0/394009962 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd0e8140110 0x7fd0e8136e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:29.821 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.814+0000 7fd0ed052640 1 --2- 172.21.15.117:0/394009962 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd0e80aa3c0 0x7fd0e81352c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:29.821 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.814+0000 7fd0ed853640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:29.821 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.814+0000 7fd0ed853640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:29.821 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.814+0000 7fd0ed853640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:29.821 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.815+0000 7fd0ee054640 10 monclient(hunting): get_auth_request con 0x7fd0e8140110 auth_method 0 2024-05-15T13:20:29.821 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.815+0000 7fd0ee054640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:29.821 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.815+0000 7fd0ee054640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:29.821 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.815+0000 7fd0ee054640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:29.821 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.815+0000 7fd0ed052640 10 monclient(hunting): get_auth_request con 0x7fd0e80aa3c0 auth_method 0 2024-05-15T13:20:29.821 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.815+0000 7fd0ed052640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:29.821 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.815+0000 7fd0ed052640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:29.821 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.815+0000 7fd0ed052640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:29.821 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.816+0000 7fd0ed052640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:29.821 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.816+0000 7fd0ed052640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:29.822 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.816+0000 7fd0ed052640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:29.824 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.817+0000 7fd0ee054640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:29.824 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.817+0000 7fd0ee054640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:29.824 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.817+0000 7fd0ee054640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:29.824 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.817+0000 7fd0ed853640 10 monclient(hunting): handle_auth_done global_id 22159 payload 824 2024-05-15T13:20:29.824 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.817+0000 7fd0ed853640 1 -- 172.21.15.117:0/394009962 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd0e80aa3c0 msgr2=0x7fd0e81352c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:29.824 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.817+0000 7fd0ed853640 1 --2- 172.21.15.117:0/394009962 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd0e80aa3c0 0x7fd0e81352c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:29.824 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.817+0000 7fd0ed853640 1 -- 172.21.15.117:0/394009962 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd0e8140110 msgr2=0x7fd0e8136e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:29.824 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.817+0000 7fd0ed853640 1 --2- 172.21.15.117:0/394009962 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd0e8140110 0x7fd0e8136e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:29.824 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.817+0000 7fd0ed853640 10 monclient: _finish_hunting 0 2024-05-15T13:20:29.824 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.817+0000 7fd0ed853640 1 monclient: found mon.noname-b 2024-05-15T13:20:29.824 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.817+0000 7fd0ed853640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:20:29.824 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.817+0000 7fd0ed853640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2024-05-15T13:20:29.825 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.817+0000 7fd0ed853640 1 -- 172.21.15.117:0/394009962 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0e8136440 con 0x7fd0e80a9b60 2024-05-15T13:20:29.825 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.820+0000 7fd0ed853640 1 --2- 172.21.15.117:0/394009962 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd0e80a9b60 0x7fd0e8146650 secure :-1 s=READY pgs=5446 cs=0 l=1 rev1=1 crypto rx=0x7fd0e000c4d0 tx=0x7fd0e000c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:29.825 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.820+0000 7fd0e67fc640 10 client.?.objecter ms_handle_connect 0x7fd0e80a9b60 2024-05-15T13:20:29.825 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.820+0000 7fd0e67fc640 10 client.?.objecter resend_mon_ops 2024-05-15T13:20:29.825 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.823+0000 7fd0e67fc640 1 -- 172.21.15.117:0/394009962 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e0017070 con 0x7fd0e80a9b60 2024-05-15T13:20:29.825 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.823+0000 7fd0e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:29.825 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.823+0000 7fd0e67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T13:20:29.825 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.823+0000 7fd0e67fc640 10 monclient: dump: 2024-05-15T13:20:29.825 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:29.825 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:29.825 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:29.826 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:29.826 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:29.826 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:29.826 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:29.826 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:29.826 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:29.827 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:29.827 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.823+0000 7fd0e67fc640 10 monclient: _finish_auth 0 2024-05-15T13:20:29.827 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.823+0000 7fd0e67fc640 10 monclient: _check_auth_tickets 2024-05-15T13:20:29.827 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.823+0000 7fd0e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:20:29.827 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.823+0000 7fd0e67fc640 1 -- 172.21.15.117:0/394009962 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0e00074e0 con 0x7fd0e80a9b60 2024-05-15T13:20:29.827 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.823+0000 7fd0efab7640 5 monclient: authenticate success, global_id 22159 2024-05-15T13:20:29.827 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.823+0000 7fd0efab7640 10 monclient: _renew_subs 2024-05-15T13:20:29.827 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.823+0000 7fd0efab7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T13:20:29.827 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.823+0000 7fd0efab7640 1 -- 172.21.15.117:0/394009962 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0e8136720 con 0x7fd0e80a9b60 2024-05-15T13:20:29.827 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.823+0000 7fd0e67fc640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:20:29.827 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.823+0000 7fd0e67fc640 1 -- 172.21.15.117:0/394009962 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e00059a0 con 0x7fd0e80a9b60 2024-05-15T13:20:29.827 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.823+0000 7fd0e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:29.828 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.823+0000 7fd0e67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T13:20:29.828 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.823+0000 7fd0e67fc640 10 monclient: dump: 2024-05-15T13:20:29.828 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:29.828 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:29.828 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:29.828 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:29.828 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:29.828 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:29.828 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:29.828 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:29.828 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:29.828 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:29.828 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.823+0000 7fd0e7fff640 4 set_mon_vals no callback set 2024-05-15T13:20:29.828 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.823+0000 7fd0efab7640 10 client.22159.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:20:29.828 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.824+0000 7fd0efab7640 10 monclient: _renew_subs 2024-05-15T13:20:29.829 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.824+0000 7fd0efab7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T13:20:29.829 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.824+0000 7fd0efab7640 1 -- 172.21.15.117:0/394009962 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0e8135800 con 0x7fd0e80a9b60 2024-05-15T13:20:29.829 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.824+0000 7fd0efab7640 1 librados: init done 2024-05-15T13:20:29.829 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.824+0000 7fd0efab7640 10 client.22159.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:20:29.829 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.824+0000 7fd0efab7640 15 client.22159.objecter _session_command_op_assign -1 1 2024-05-15T13:20:29.829 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.824+0000 7fd0efab7640 20 client.22159.objecter _get_session osd=-1 returning homeless 2024-05-15T13:20:29.829 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.824+0000 7fd0efab7640 10 client.22159.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:20:29.829 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.824+0000 7fd0efab7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T13:20:29.829 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.824+0000 7fd0efab7640 1 -- 172.21.15.117:0/394009962 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd0e80b2f40 con 0x7fd0e80a9b60 2024-05-15T13:20:29.845 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.845+0000 7fd0e67fc640 1 -- 172.21.15.117:0/394009962 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fd0e001a070 con 0x7fd0e80a9b60 2024-05-15T13:20:29.845 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.846+0000 7fd0e67fc640 1 --2- 172.21.15.117:0/394009962 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fd0d406cee0 0x7fd0d406f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:29.845 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.846+0000 7fd0ed052640 1 --2- 172.21.15.117:0/394009962 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fd0d406cee0 0x7fd0d406f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:29.846 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.846+0000 7fd0ed052640 10 monclient: get_auth_request con 0x7fd0d406cee0 auth_method 0 2024-05-15T13:20:29.846 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.846+0000 7fd0e67fc640 10 client.22159.objecter ms_handle_connect 0x7fd0d406cee0 2024-05-15T13:20:29.846 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.846+0000 7fd0ed052640 1 --2- 172.21.15.117:0/394009962 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fd0d406cee0 0x7fd0d406f3a0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fd0e813d560 tx=0x7fd0dc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:29.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.849+0000 7fd0e67fc640 1 -- 172.21.15.117:0/394009962 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(1225..1225 src has 912..1225) v4 ==== 6365+0+0 (secure 0 0 0) 0x7fd0e0056910 con 0x7fd0e80a9b60 2024-05-15T13:20:29.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.849+0000 7fd0e67fc640 10 client.22159.objecter ms_dispatch 0x7fd0e8000b90 osd_map(1225..1225 src has 912..1225) v4 2024-05-15T13:20:29.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.849+0000 7fd0e67fc640 3 client.22159.objecter handle_osd_map got epochs [1225,1225] > 0 2024-05-15T13:20:29.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.849+0000 7fd0e67fc640 3 client.22159.objecter handle_osd_map decoding full epoch 1225 2024-05-15T13:20:29.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.849+0000 7fd0e67fc640 10 client.22159.objecter checking command 1 2024-05-15T13:20:29.850 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.849+0000 7fd0e67fc640 1 --2- 172.21.15.117:0/394009962 >> [v2:172.21.15.117:6804/1425835241,v1:172.21.15.117:6805/1425835241] conn(0x7fd0d4074bf0 0x7fd0d4077090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:29.850 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.849+0000 7fd0e67fc640 20 client.22159.objecter _get_session s=0x7fd0d4074520 osd=2 3 2024-05-15T13:20:29.850 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.849+0000 7fd0e67fc640 20 client.22159.objecter put_session s=0x7fd0d4074520 osd=2 3 2024-05-15T13:20:29.850 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.849+0000 7fd0e67fc640 15 client.22159.objecter _session_command_op_remove -1 1 2024-05-15T13:20:29.850 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.849+0000 7fd0e67fc640 20 client.22159.objecter _get_session s=0x7fd0d4074520 osd=2 3 2024-05-15T13:20:29.850 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.849+0000 7fd0e67fc640 20 client.22159.objecter get_session s=0x7fd0d4074520 osd=2 3 2024-05-15T13:20:29.850 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.849+0000 7fd0e67fc640 15 client.22159.objecter _session_command_op_assign 2 1 2024-05-15T13:20:29.850 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.849+0000 7fd0e67fc640 20 client.22159.objecter put_session s=0x7fd0d4074520 osd=2 4 2024-05-15T13:20:29.850 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.849+0000 7fd0e67fc640 10 client.22159.objecter _send_command 1 2024-05-15T13:20:29.851 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.849+0000 7fd0e67fc640 1 -- 172.21.15.117:0/394009962 --> [v2:172.21.15.117:6804/1425835241,v1:172.21.15.117:6805/1425835241] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd0e0024ea0 con 0x7fd0d4074bf0 2024-05-15T13:20:29.851 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.849+0000 7fd0e67fc640 20 client.22159.objecter dump_active .. 0 homeless 2024-05-15T13:20:29.851 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.850+0000 7fd0ee054640 1 --2- 172.21.15.117:0/394009962 >> [v2:172.21.15.117:6804/1425835241,v1:172.21.15.117:6805/1425835241] conn(0x7fd0d4074bf0 0x7fd0d4077090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:29.851 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.850+0000 7fd0ee054640 10 monclient: get_auth_request con 0x7fd0d4074bf0 auth_method 0 2024-05-15T13:20:29.852 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.853+0000 7fd0e67fc640 10 client.22159.objecter ms_handle_connect 0x7fd0d4074bf0 2024-05-15T13:20:29.852 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.853+0000 7fd0ee054640 1 --2- 172.21.15.117:0/394009962 >> [v2:172.21.15.117:6804/1425835241,v1:172.21.15.117:6805/1425835241] conn(0x7fd0d4074bf0 0x7fd0d4077090 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:29.879 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.880+0000 7fd0e67fc640 1 -- 172.21.15.117:0/394009962 <== osd.2 v2:172.21.15.117:6804/1425835241 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd0e0024ea0 con 0x7fd0d4074bf0 2024-05-15T13:20:29.879 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.880+0000 7fd0e67fc640 10 client.22159.objecter ms_dispatch 0x7fd0e8000b90 command_reply(tid 1: 0 ) v1 2024-05-15T13:20:29.879 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.880+0000 7fd0e67fc640 10 client.22159.objecter _finish_command 1 = system:0 2024-05-15T13:20:29.879 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.880+0000 7fd0e67fc640 20 client.22159.objecter put_session s=0x7fd0d4074520 osd=2 4 2024-05-15T13:20:29.879 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.880+0000 7fd0e67fc640 15 client.22159.objecter _session_command_op_remove 2 1 2024-05-15T13:20:29.893 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.895+0000 7fd0efab7640 10 client.22159.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-05-15T13:20:29.893 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.895+0000 7fd0efab7640 15 client.22159.objecter _session_command_op_assign -1 2 2024-05-15T13:20:29.893 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.895+0000 7fd0efab7640 20 client.22159.objecter _get_session s=0x7fd0d4074520 osd=2 3 2024-05-15T13:20:29.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.895+0000 7fd0efab7640 20 client.22159.objecter put_session s=0x7fd0d4074520 osd=2 3 2024-05-15T13:20:29.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.895+0000 7fd0efab7640 20 client.22159.objecter _get_session s=0x7fd0d4074520 osd=2 3 2024-05-15T13:20:29.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.895+0000 7fd0efab7640 15 client.22159.objecter _session_command_op_remove -1 2 2024-05-15T13:20:29.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.895+0000 7fd0efab7640 20 client.22159.objecter get_session s=0x7fd0d4074520 osd=2 3 2024-05-15T13:20:29.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.895+0000 7fd0efab7640 15 client.22159.objecter _session_command_op_assign 2 2 2024-05-15T13:20:29.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.895+0000 7fd0efab7640 20 client.22159.objecter put_session s=0x7fd0d4074520 osd=2 4 2024-05-15T13:20:29.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.895+0000 7fd0efab7640 10 client.22159.objecter _send_command 2 2024-05-15T13:20:29.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.895+0000 7fd0efab7640 1 -- 172.21.15.117:0/394009962 --> [v2:172.21.15.117:6804/1425835241,v1:172.21.15.117:6805/1425835241] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fd0e804f0e0 con 0x7fd0d4074bf0 2024-05-15T13:20:29.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.902+0000 7fd0e67fc640 1 -- 172.21.15.117:0/394009962 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_get_version_reply(handle=1 version=1225) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd0e0054b10 con 0x7fd0e80a9b60 2024-05-15T13:20:29.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.902+0000 7fd0e67fc640 10 monclient: handle_get_version_reply finishing 1 version 1225 2024-05-15T13:20:29.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.911+0000 7fd0e67fc640 1 -- 172.21.15.117:0/394009962 <== osd.2 v2:172.21.15.117:6804/1425835241 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fd0e804f0e0 con 0x7fd0d4074bf0 2024-05-15T13:20:29.910 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.911+0000 7fd0e67fc640 10 client.22159.objecter ms_dispatch 0x7fd0e8000b90 command_reply(tid 2: 0 ) v1 2024-05-15T13:20:29.910 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.911+0000 7fd0e67fc640 10 client.22159.objecter _finish_command 2 = system:0 2024-05-15T13:20:29.910 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.911+0000 7fd0e67fc640 20 client.22159.objecter put_session s=0x7fd0d4074520 osd=2 4 2024-05-15T13:20:29.910 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.911+0000 7fd0e67fc640 15 client.22159.objecter _session_command_op_remove 2 2 2024-05-15T13:20:29.910 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-05-15T13:20:29.910 INFO:teuthology.orchestra.run.smithi117.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-15T13:20:29.910 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-05-15T13:20:29.912 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.911+0000 7fd0efab7640 10 librados: watch_flush enter 2024-05-15T13:20:29.912 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.911+0000 7fd0efab7640 10 librados: watch_flush exit 2024-05-15T13:20:29.912 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.911+0000 7fd0efab7640 10 client.22159.objecter close_session for osd.2 2024-05-15T13:20:29.912 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.911+0000 7fd0efab7640 1 -- 172.21.15.117:0/394009962 >> [v2:172.21.15.117:6804/1425835241,v1:172.21.15.117:6805/1425835241] conn(0x7fd0d4074bf0 msgr2=0x7fd0d4077090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:29.912 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.911+0000 7fd0efab7640 1 --2- 172.21.15.117:0/394009962 >> [v2:172.21.15.117:6804/1425835241,v1:172.21.15.117:6805/1425835241] conn(0x7fd0d4074bf0 0x7fd0d4077090 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:29.912 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.911+0000 7fd0efab7640 20 client.22159.objecter put_session s=0x7fd0d4074520 osd=2 1 2024-05-15T13:20:29.912 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.911+0000 7fd0efab7640 20 client.22159.objecter shutdown clearing up homeless session... 2024-05-15T13:20:29.912 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.911+0000 7fd0efab7640 10 client.22159.objecter successfully canceled tick 2024-05-15T13:20:29.912 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.911+0000 7fd0efab7640 1 -- 172.21.15.117:0/394009962 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fd0d406cee0 msgr2=0x7fd0d406f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:29.912 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.911+0000 7fd0efab7640 1 --2- 172.21.15.117:0/394009962 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fd0d406cee0 0x7fd0d406f3a0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fd0e813d560 tx=0x7fd0dc009040 comp rx=0 tx=0).stop 2024-05-15T13:20:29.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.911+0000 7fd0efab7640 10 monclient: shutdown 2024-05-15T13:20:29.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.911+0000 7fd0efab7640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:20:29.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.911+0000 7fd0efab7640 1 -- 172.21.15.117:0/394009962 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd0e80a9b60 msgr2=0x7fd0e8146650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:29.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.911+0000 7fd0efab7640 1 --2- 172.21.15.117:0/394009962 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd0e80a9b60 0x7fd0e8146650 secure :-1 s=READY pgs=5446 cs=0 l=1 rev1=1 crypto rx=0x7fd0e000c4d0 tx=0x7fd0e000c8d0 comp rx=0 tx=0).stop 2024-05-15T13:20:29.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.911+0000 7fd0ee054640 1 -- 172.21.15.117:0/394009962 reap_dead start 2024-05-15T13:20:29.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.911+0000 7fd0efab7640 1 -- 172.21.15.117:0/394009962 shutdown_connections 2024-05-15T13:20:29.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.911+0000 7fd0efab7640 1 -- 172.21.15.117:0/394009962 >> 172.21.15.117:0/394009962 conn(0x7fd0e80b7f90 msgr2=0x7fd0e80b2830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:20:29.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.911+0000 7fd0efab7640 1 -- 172.21.15.117:0/394009962 shutdown_connections 2024-05-15T13:20:29.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.911+0000 7fd0efab7640 1 -- 172.21.15.117:0/394009962 wait complete. 2024-05-15T13:20:29.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:29.911+0000 7fd0efab7640 1 librados: shutdown 2024-05-15T13:20:29.921 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-05-15T13:20:30.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.023+0000 7fb5eb1af640 10 monclient: get_monmap_and_config 2024-05-15T13:20:30.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.023+0000 7fb5eb1af640 10 monclient: build_initial_monmap 2024-05-15T13:20:30.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.023+0000 7fb5eb1af640 10 monclient: monmap: 2024-05-15T13:20:30.022 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T13:20:30.022 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:30.023 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T13:20:30.024837+0000 2024-05-15T13:20:30.023 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T13:20:30.024837+0000 2024-05-15T13:20:30.023 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T13:20:30.023 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:30.023 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T13:20:30.023 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T13:20:30.023 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T13:20:30.023 INFO:teuthology.orchestra.run.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T13:20:30.023 INFO:teuthology.orchestra.run.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T13:20:30.023 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:30.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.024+0000 7fb5eb1af640 1 Processor -- start 2024-05-15T13:20:30.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.024+0000 7fb5eb1af640 1 -- start start 2024-05-15T13:20:30.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.024+0000 7fb5eb1af640 10 monclient: init 2024-05-15T13:20:30.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.025+0000 7fb5eb1af640 10 monclient: _reopen_session rank -1 2024-05-15T13:20:30.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.025+0000 7fb5eb1af640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-05-15T13:20:30.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.025+0000 7fb5eb1af640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fb5e40b86e0 0x7fb5e40aa0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:30.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.025+0000 7fb5eb1af640 10 monclient(hunting): picked mon.noname-c con 0x7fb5e40b86e0 addr v2:172.21.15.178:3300/0 2024-05-15T13:20:30.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.025+0000 7fb5eb1af640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb5e40b1dd0 addr v1:172.21.15.117:6789/0 2024-05-15T13:20:30.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.025+0000 7fb5eb1af640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fb5e413f870 0x7fb5e413fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:30.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.025+0000 7fb5eb1af640 10 monclient(hunting): picked mon.noname-a con 0x7fb5e413f870 addr v2:172.21.15.117:3300/0 2024-05-15T13:20:30.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.025+0000 7fb5eb1af640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5e40a9bc0 con 0x7fb5e40b1dd0 2024-05-15T13:20:30.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.025+0000 7fb5eb1af640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:30.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.025+0000 7fb5eb1af640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5e40b8ac0 con 0x7fb5e413f870 2024-05-15T13:20:30.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.025+0000 7fb5eb1af640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:30.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.025+0000 7fb5eb1af640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5e40aa620 con 0x7fb5e40b86e0 2024-05-15T13:20:30.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.025+0000 7fb5eb1af640 10 monclient(hunting): _renew_subs 2024-05-15T13:20:30.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.025+0000 7fb5eb1af640 10 monclient(hunting): authenticate will time out at 3593.267093s 2024-05-15T13:20:30.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.025+0000 7fb5e8f4b640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fb5e40b86e0 0x7fb5e40aa0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:30.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.026+0000 7fb5e974c640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fb5e413f870 0x7fb5e413fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:30.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.026+0000 7fb5e3fff640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fb5e40b1dd0 0x7fb5e4145bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:41454/0 (socket says 172.21.15.117:41454) 2024-05-15T13:20:30.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.026+0000 7fb5e3fff640 1 -- 172.21.15.117:0/3491913922 learned_addr learned my addr 172.21.15.117:0/3491913922 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-05-15T13:20:30.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.026+0000 7fb5e8f4b640 10 monclient(hunting): get_auth_request con 0x7fb5e40b86e0 auth_method 0 2024-05-15T13:20:30.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.026+0000 7fb5e8f4b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:30.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.026+0000 7fb5e8f4b640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:30.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.026+0000 7fb5e8f4b640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:30.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.026+0000 7fb5e974c640 10 monclient(hunting): get_auth_request con 0x7fb5e413f870 auth_method 0 2024-05-15T13:20:30.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.026+0000 7fb5e974c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:30.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.026+0000 7fb5e974c640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:30.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.026+0000 7fb5e974c640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:30.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.026+0000 7fb5e8f4b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:30.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.026+0000 7fb5e8f4b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:30.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.026+0000 7fb5e8f4b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:30.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.027+0000 7fb5e974c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:30.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.027+0000 7fb5e974c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:30.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.027+0000 7fb5e974c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:30.029 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.028+0000 7fb5e8f4b640 10 monclient(hunting): handle_auth_done global_id 22162 payload 274 2024-05-15T13:20:30.029 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.028+0000 7fb5e8f4b640 1 -- 172.21.15.117:0/3491913922 >> v2:172.21.15.117:3300/0 conn(0x7fb5e413f870 msgr2=0x7fb5e413fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T13:20:30.029 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.028+0000 7fb5e8f4b640 1 --2- 172.21.15.117:0/3491913922 >> v2:172.21.15.117:3300/0 conn(0x7fb5e413f870 0x7fb5e413fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:30.029 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.028+0000 7fb5e8f4b640 1 -- 172.21.15.117:0/3491913922 >> v1:172.21.15.117:6789/0 conn(0x7fb5e40b1dd0 legacy=0x7fb5e4145bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:30.029 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.028+0000 7fb5e8f4b640 10 monclient: _finish_hunting 0 2024-05-15T13:20:30.029 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.028+0000 7fb5e8f4b640 1 monclient: found mon.noname-c 2024-05-15T13:20:30.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.028+0000 7fb5e8f4b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:20:30.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.028+0000 7fb5e8f4b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.178:3300/0 2024-05-15T13:20:30.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.028+0000 7fb5e8f4b640 1 -- 172.21.15.117:0/3491913922 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e40a9850 con 0x7fb5e40b86e0 2024-05-15T13:20:30.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.030+0000 7fb5e8f4b640 1 --2- 172.21.15.117:0/3491913922 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb5e40b86e0 0x7fb5e40aa0e0 secure :-1 s=READY pgs=5447 cs=0 l=1 rev1=1 crypto rx=0x7fb5e40b8d50 tx=0x7fb5d402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9fa49c296180cfcb server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:30.032 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.034+0000 7fb5e37fe640 1 -- 172.21.15.117:0/3491913922 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d40315a0 con 0x7fb5e40b86e0 2024-05-15T13:20:30.032 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.034+0000 7fb5e37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:30.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.034+0000 7fb5e37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T13:20:30.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.034+0000 7fb5e37fe640 10 monclient: dump: 2024-05-15T13:20:30.033 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:30.033 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:30.033 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:30.033 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:30.033 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:30.033 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:30.033 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:30.033 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:30.033 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:30.033 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:30.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.034+0000 7fb5e37fe640 10 monclient: _finish_auth 0 2024-05-15T13:20:30.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.034+0000 7fb5e37fe640 10 monclient: _check_auth_tickets 2024-05-15T13:20:30.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.034+0000 7fb5e37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:20:30.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.034+0000 7fb5e37fe640 1 -- 172.21.15.117:0/3491913922 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5d4037070 con 0x7fb5e40b86e0 2024-05-15T13:20:30.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.034+0000 7fb5eb1af640 5 monclient: authenticate success, global_id 22162 2024-05-15T13:20:30.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.034+0000 7fb5eb1af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T13:20:30.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.034+0000 7fb5e37fe640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:20:30.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.034+0000 7fb5eb1af640 10 monclient: get_monmap_and_config success 2024-05-15T13:20:30.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.034+0000 7fb5e37fe640 1 -- 172.21.15.117:0/3491913922 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d4031960 con 0x7fb5e40b86e0 2024-05-15T13:20:30.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.034+0000 7fb5e37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:30.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.034+0000 7fb5eb1af640 4 set_mon_vals no callback set 2024-05-15T13:20:30.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.034+0000 7fb5e37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T13:20:30.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.034+0000 7fb5e37fe640 10 monclient: dump: 2024-05-15T13:20:30.034 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:30.034 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:30.034 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:30.034 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:30.034 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:30.035 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:30.035 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:30.035 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:30.035 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:30.035 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:30.038 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.034+0000 7fb5eb1af640 10 monclient: shutdown 2024-05-15T13:20:30.039 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.034+0000 7fb5eb1af640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:20:30.039 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.034+0000 7fb5eb1af640 1 -- 172.21.15.117:0/3491913922 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb5e40b86e0 msgr2=0x7fb5e40aa0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:30.039 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.034+0000 7fb5eb1af640 1 --2- 172.21.15.117:0/3491913922 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb5e40b86e0 0x7fb5e40aa0e0 secure :-1 s=READY pgs=5447 cs=0 l=1 rev1=1 crypto rx=0x7fb5e40b8d50 tx=0x7fb5d402b840 comp rx=0 tx=0).stop 2024-05-15T13:20:30.039 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.034+0000 7fb5eb1af640 1 -- 172.21.15.117:0/3491913922 shutdown_connections 2024-05-15T13:20:30.039 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.034+0000 7fb5eb1af640 1 --2- 172.21.15.117:0/3491913922 >> v2:172.21.15.117:3300/0 conn(0x7fb5e413f870 0x7fb5e413fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:30.039 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.034+0000 7fb5eb1af640 1 --2- 172.21.15.117:0/3491913922 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb5e40b86e0 0x7fb5e40aa0e0 unknown :-1 s=CLOSED pgs=5447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:30.039 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.034+0000 7fb5eb1af640 1 -- 172.21.15.117:0/3491913922 >> 172.21.15.117:0/3491913922 conn(0x7fb5e4066f00 msgr2=0x7fb5e4067310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:20:30.039 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.034+0000 7fb5eb1af640 1 -- 172.21.15.117:0/3491913922 shutdown_connections 2024-05-15T13:20:30.039 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.034+0000 7fb5eb1af640 1 -- 172.21.15.117:0/3491913922 wait complete. 2024-05-15T13:20:30.039 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.035+0000 7fb5eb1af640 10 monclient: build_initial_monmap 2024-05-15T13:20:30.039 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.035+0000 7fb5eb1af640 10 monclient: monmap: 2024-05-15T13:20:30.039 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T13:20:30.039 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T13:20:30.039 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-05-15T13:20:30.040 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-05-15T13:20:30.040 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T13:20:30.040 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:30.040 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T13:20:30.040 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T13:20:30.040 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T13:20:30.040 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:30.040 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.035+0000 7fb5eb1af640 1 librados: starting msgr at 2024-05-15T13:20:30.040 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.035+0000 7fb5eb1af640 1 librados: starting objecter 2024-05-15T13:20:30.040 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.035+0000 7fb5eb1af640 1 Processor -- start 2024-05-15T13:20:30.040 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.036+0000 7fb5eb1af640 1 -- start start 2024-05-15T13:20:30.040 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.036+0000 7fb5eb1af640 1 librados: setting wanted keys 2024-05-15T13:20:30.040 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.036+0000 7fb5eb1af640 1 librados: calling monclient init 2024-05-15T13:20:30.040 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.036+0000 7fb5eb1af640 10 monclient: init 2024-05-15T13:20:30.040 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.036+0000 7fb5eb1af640 10 monclient: _reopen_session rank -1 2024-05-15T13:20:30.041 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.036+0000 7fb5eb1af640 10 monclient: _add_conns ranks=[1,2,0] 2024-05-15T13:20:30.041 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.036+0000 7fb5eb1af640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb5e40b1dd0 0x7fb5e41db830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:30.041 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.036+0000 7fb5eb1af640 10 monclient(hunting): picked mon.noname-b con 0x7fb5e40b1dd0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T13:20:30.041 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.036+0000 7fb5eb1af640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb5e40b86e0 0x7fb5e41dbd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:30.041 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.036+0000 7fb5eb1af640 10 monclient(hunting): picked mon.noname-c con 0x7fb5e40b86e0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T13:20:30.041 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.036+0000 7fb5eb1af640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb5e413f870 0x7fb5e41dc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:30.041 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.036+0000 7fb5eb1af640 10 monclient(hunting): picked mon.noname-a con 0x7fb5e413f870 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T13:20:30.041 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.036+0000 7fb5eb1af640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:30.041 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.036+0000 7fb5eb1af640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e40b3be0 con 0x7fb5e413f870 2024-05-15T13:20:30.041 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.036+0000 7fb5eb1af640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:30.041 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.036+0000 7fb5eb1af640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e40b3a60 con 0x7fb5e40b1dd0 2024-05-15T13:20:30.041 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.036+0000 7fb5eb1af640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:30.041 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.036+0000 7fb5eb1af640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e414cf10 con 0x7fb5e40b86e0 2024-05-15T13:20:30.041 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.036+0000 7fb5eb1af640 10 monclient(hunting): _renew_subs 2024-05-15T13:20:30.042 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.036+0000 7fb5eb1af640 10 monclient(hunting): authenticate will time out at 3593.278549s 2024-05-15T13:20:30.042 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.037+0000 7fb5e8f4b640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb5e40b1dd0 0x7fb5e41db830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:30.042 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.037+0000 7fb5e974c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb5e413f870 0x7fb5e41dc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:30.042 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.037+0000 7fb5e3fff640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb5e40b86e0 0x7fb5e41dbd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:30.042 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.037+0000 7fb5e8f4b640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb5e40b1dd0 0x7fb5e41db830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:37306/0 (socket says 172.21.15.117:37306) 2024-05-15T13:20:30.042 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.037+0000 7fb5e8f4b640 1 -- 172.21.15.117:0/188357588 learned_addr learned my addr 172.21.15.117:0/188357588 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T13:20:30.042 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.037+0000 7fb5e8f4b640 10 monclient(hunting): get_auth_request con 0x7fb5e40b1dd0 auth_method 0 2024-05-15T13:20:30.042 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.037+0000 7fb5e8f4b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:30.042 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.037+0000 7fb5e8f4b640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:30.042 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.037+0000 7fb5e8f4b640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:30.042 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.037+0000 7fb5e974c640 10 monclient(hunting): get_auth_request con 0x7fb5e413f870 auth_method 0 2024-05-15T13:20:30.042 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.037+0000 7fb5e974c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:30.042 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.037+0000 7fb5e974c640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:30.042 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.037+0000 7fb5e974c640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:30.042 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.037+0000 7fb5e3fff640 10 monclient(hunting): get_auth_request con 0x7fb5e40b86e0 auth_method 0 2024-05-15T13:20:30.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.037+0000 7fb5e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:30.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.037+0000 7fb5e3fff640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:30.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.037+0000 7fb5e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:30.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.037+0000 7fb5e8f4b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:30.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.037+0000 7fb5e8f4b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:30.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.037+0000 7fb5e8f4b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:30.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.038+0000 7fb5e974c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:30.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.038+0000 7fb5e974c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:30.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.038+0000 7fb5e974c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:30.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.038+0000 7fb5e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:30.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.038+0000 7fb5e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:30.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.038+0000 7fb5e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:30.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.040+0000 7fb5e8f4b640 10 monclient(hunting): handle_auth_done global_id 22165 payload 824 2024-05-15T13:20:30.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.040+0000 7fb5e8f4b640 1 -- 172.21.15.117:0/188357588 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb5e40b86e0 msgr2=0x7fb5e41dbd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:30.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.040+0000 7fb5e8f4b640 1 --2- 172.21.15.117:0/188357588 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb5e40b86e0 0x7fb5e41dbd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:30.044 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.040+0000 7fb5e8f4b640 1 -- 172.21.15.117:0/188357588 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb5e413f870 msgr2=0x7fb5e41dc2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:30.044 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.040+0000 7fb5e8f4b640 1 --2- 172.21.15.117:0/188357588 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb5e413f870 0x7fb5e41dc2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:30.044 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.040+0000 7fb5e8f4b640 10 monclient: _finish_hunting 0 2024-05-15T13:20:30.044 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.040+0000 7fb5e8f4b640 1 monclient: found mon.noname-b 2024-05-15T13:20:30.044 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.040+0000 7fb5e8f4b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:20:30.044 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.040+0000 7fb5e8f4b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2024-05-15T13:20:30.044 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.040+0000 7fb5e8f4b640 1 -- 172.21.15.117:0/188357588 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e4143330 con 0x7fb5e40b1dd0 2024-05-15T13:20:30.046 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.042+0000 7fb5e8f4b640 1 --2- 172.21.15.117:0/188357588 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb5e40b1dd0 0x7fb5e41db830 secure :-1 s=READY pgs=5448 cs=0 l=1 rev1=1 crypto rx=0x7fb5d4009890 tx=0x7fb5d40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:30.046 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.042+0000 7fb5e17fa640 10 client.?.objecter ms_handle_connect 0x7fb5e40b1dd0 2024-05-15T13:20:30.046 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.042+0000 7fb5e17fa640 10 client.?.objecter resend_mon_ops 2024-05-15T13:20:30.046 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.046+0000 7fb5e17fa640 1 -- 172.21.15.117:0/188357588 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d4032630 con 0x7fb5e40b1dd0 2024-05-15T13:20:30.046 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.046+0000 7fb5e17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:30.046 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.046+0000 7fb5e17fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T13:20:30.046 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.046+0000 7fb5e17fa640 10 monclient: dump: 2024-05-15T13:20:30.046 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:30.046 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:30.047 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:30.047 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:30.047 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:30.047 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:30.047 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:30.047 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:30.047 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:30.047 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:30.047 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.046+0000 7fb5e17fa640 10 monclient: _finish_auth 0 2024-05-15T13:20:30.047 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.046+0000 7fb5e17fa640 10 monclient: _check_auth_tickets 2024-05-15T13:20:30.047 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.046+0000 7fb5e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:20:30.047 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.046+0000 7fb5e17fa640 1 -- 172.21.15.117:0/188357588 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5d40327d0 con 0x7fb5e40b1dd0 2024-05-15T13:20:30.047 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.046+0000 7fb5eb1af640 5 monclient: authenticate success, global_id 22165 2024-05-15T13:20:30.047 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.046+0000 7fb5e17fa640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:20:30.048 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.046+0000 7fb5e17fa640 1 -- 172.21.15.117:0/188357588 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d4032bc0 con 0x7fb5e40b1dd0 2024-05-15T13:20:30.048 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.046+0000 7fb5eb1af640 10 monclient: _renew_subs 2024-05-15T13:20:30.048 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.046+0000 7fb5eb1af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T13:20:30.048 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.046+0000 7fb5eb1af640 1 -- 172.21.15.117:0/188357588 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5e41427e0 con 0x7fb5e40b1dd0 2024-05-15T13:20:30.048 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.046+0000 7fb5e17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:30.048 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.046+0000 7fb5e17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T13:20:30.048 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.046+0000 7fb5e17fa640 10 monclient: dump: 2024-05-15T13:20:30.048 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:30.048 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:30.048 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:30.048 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:30.048 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:30.048 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:30.048 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:30.048 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:30.049 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:30.049 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:30.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.046+0000 7fb5e2ffd640 4 set_mon_vals no callback set 2024-05-15T13:20:30.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.046+0000 7fb5eb1af640 10 client.22165.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:20:30.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.046+0000 7fb5eb1af640 10 monclient: _renew_subs 2024-05-15T13:20:30.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.046+0000 7fb5eb1af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T13:20:30.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.046+0000 7fb5eb1af640 1 -- 172.21.15.117:0/188357588 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5e413c9d0 con 0x7fb5e40b1dd0 2024-05-15T13:20:30.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.046+0000 7fb5eb1af640 1 librados: init done 2024-05-15T13:20:30.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.047+0000 7fb5eb1af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T13:20:30.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.047+0000 7fb5eb1af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:20:30.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.047+0000 7fb5eb1af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T13:20:30.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.047+0000 7fb5eb1af640 1 -- 172.21.15.117:0/188357588 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5e404f0e0 con 0x7fb5e40b1dd0 2024-05-15T13:20:30.064 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.065+0000 7fb5e17fa640 1 -- 172.21.15.117:0/188357588 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fb5d404e430 con 0x7fb5e40b1dd0 2024-05-15T13:20:30.064 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.065+0000 7fb5e17fa640 1 --2- 172.21.15.117:0/188357588 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fb5d006ceb0 0x7fb5d006f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:30.064 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.066+0000 7fb5e3fff640 1 --2- 172.21.15.117:0/188357588 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fb5d006ceb0 0x7fb5d006f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:30.065 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.066+0000 7fb5e3fff640 10 monclient: get_auth_request con 0x7fb5d006ceb0 auth_method 0 2024-05-15T13:20:30.065 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.066+0000 7fb5e3fff640 1 --2- 172.21.15.117:0/188357588 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fb5d006ceb0 0x7fb5d006f370 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fb5e41dc8f0 tx=0x7fb5d8007e80 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:30.065 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.066+0000 7fb5e17fa640 10 client.22165.objecter ms_handle_connect 0x7fb5d006ceb0 2024-05-15T13:20:30.066 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.067+0000 7fb5e17fa640 1 -- 172.21.15.117:0/188357588 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(1225..1225 src has 912..1225) v4 ==== 6365+0+0 (secure 0 0 0) 0x7fb5d402f070 con 0x7fb5e40b1dd0 2024-05-15T13:20:30.066 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.067+0000 7fb5e17fa640 10 client.22165.objecter ms_dispatch 0x7fb5e4000b90 osd_map(1225..1225 src has 912..1225) v4 2024-05-15T13:20:30.066 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.067+0000 7fb5e17fa640 3 client.22165.objecter handle_osd_map got epochs [1225,1225] > 0 2024-05-15T13:20:30.066 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.067+0000 7fb5e17fa640 3 client.22165.objecter handle_osd_map decoding full epoch 1225 2024-05-15T13:20:30.066 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.067+0000 7fb5e17fa640 20 client.22165.objecter dump_active .. 0 homeless 2024-05-15T13:20:30.304 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.305+0000 7fb5e17fa640 1 -- 172.21.15.117:0/188357588 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fb5d4040070 con 0x7fb5e40b1dd0 2024-05-15T13:20:30.304 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.305+0000 7fb5e17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:20:30.304 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.305+0000 7fb5e17fa640 10 monclient: _finish_command 1 = system:0 2024-05-15T13:20:30.498 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.499+0000 7fb5eb1af640 1 -- 172.21.15.117:0/188357588 --> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7fb5e4142fb0 con 0x7fb5d006ceb0 2024-05-15T13:20:30.498 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.500+0000 7fb5e17fa640 1 -- 172.21.15.117:0/188357588 <== mgr.4107 v2:172.21.15.117:6800/1785766523 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb5e4142fb0 con 0x7fb5d006ceb0 2024-05-15T13:20:30.499 INFO:teuthology.orchestra.run.smithi117.stderr:instructed osd(s) 2 to deep-scrub 2024-05-15T13:20:30.501 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.501+0000 7fb5eb1af640 10 librados: watch_flush enter 2024-05-15T13:20:30.501 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.501+0000 7fb5eb1af640 10 librados: watch_flush exit 2024-05-15T13:20:30.501 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.501+0000 7fb5eb1af640 20 client.22165.objecter shutdown clearing up homeless session... 2024-05-15T13:20:30.501 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.501+0000 7fb5eb1af640 10 client.22165.objecter successfully canceled tick 2024-05-15T13:20:30.501 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.502+0000 7fb5eb1af640 1 -- 172.21.15.117:0/188357588 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fb5d006ceb0 msgr2=0x7fb5d006f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:30.501 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.502+0000 7fb5eb1af640 1 --2- 172.21.15.117:0/188357588 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fb5d006ceb0 0x7fb5d006f370 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fb5e41dc8f0 tx=0x7fb5d8007e80 comp rx=0 tx=0).stop 2024-05-15T13:20:30.502 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.502+0000 7fb5eb1af640 10 monclient: shutdown 2024-05-15T13:20:30.502 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.502+0000 7fb5eb1af640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:20:30.502 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.502+0000 7fb5eb1af640 1 -- 172.21.15.117:0/188357588 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb5e40b1dd0 msgr2=0x7fb5e41db830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:30.502 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.502+0000 7fb5eb1af640 1 --2- 172.21.15.117:0/188357588 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb5e40b1dd0 0x7fb5e41db830 secure :-1 s=READY pgs=5448 cs=0 l=1 rev1=1 crypto rx=0x7fb5d4009890 tx=0x7fb5d40046c0 comp rx=0 tx=0).stop 2024-05-15T13:20:30.502 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.502+0000 7fb5eb1af640 1 -- 172.21.15.117:0/188357588 shutdown_connections 2024-05-15T13:20:30.502 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.502+0000 7fb5eb1af640 1 --2- 172.21.15.117:0/188357588 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb5e413f870 0x7fb5e41dc2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:30.502 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.502+0000 7fb5eb1af640 1 --2- 172.21.15.117:0/188357588 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb5e40b86e0 0x7fb5e41dbd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:30.502 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.502+0000 7fb5eb1af640 1 --2- 172.21.15.117:0/188357588 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fb5d006ceb0 0x7fb5d006f370 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:30.502 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.502+0000 7fb5eb1af640 1 --2- 172.21.15.117:0/188357588 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb5e40b1dd0 0x7fb5e41db830 unknown :-1 s=CLOSED pgs=5448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:30.502 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.502+0000 7fb5eb1af640 1 -- 172.21.15.117:0/188357588 >> 172.21.15.117:0/188357588 conn(0x7fb5e4066f00 msgr2=0x7fb5e40b4740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:20:30.502 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.502+0000 7fb5eb1af640 1 -- 172.21.15.117:0/188357588 shutdown_connections 2024-05-15T13:20:30.502 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.502+0000 7fb5eb1af640 1 -- 172.21.15.117:0/188357588 wait complete. 2024-05-15T13:20:30.502 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.502+0000 7fb5eb1af640 1 librados: shutdown 2024-05-15T13:20:30.512 INFO:tasks.ceph:Scrubbing osd.3 2024-05-15T13:20:30.512 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-05-15T13:20:30.623 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.624+0000 7f09572be640 10 monclient: get_monmap_and_config 2024-05-15T13:20:30.623 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.624+0000 7f09572be640 10 monclient: build_initial_monmap 2024-05-15T13:20:30.623 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.624+0000 7f09572be640 10 monclient: monmap: 2024-05-15T13:20:30.623 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T13:20:30.623 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:30.623 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T13:20:30.625793+0000 2024-05-15T13:20:30.624 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T13:20:30.625793+0000 2024-05-15T13:20:30.624 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T13:20:30.624 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:30.624 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T13:20:30.624 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T13:20:30.624 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T13:20:30.624 INFO:teuthology.orchestra.run.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T13:20:30.624 INFO:teuthology.orchestra.run.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T13:20:30.624 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:30.626 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.625+0000 7f09572be640 1 Processor -- start 2024-05-15T13:20:30.626 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.625+0000 7f09572be640 1 -- start start 2024-05-15T13:20:30.626 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.625+0000 7f09572be640 10 monclient: init 2024-05-15T13:20:30.626 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.626+0000 7f09572be640 10 monclient: _reopen_session rank -1 2024-05-15T13:20:30.626 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.626+0000 7f09572be640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-05-15T13:20:30.626 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.626+0000 7f09572be640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f095006c900 0x7f09500aeab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:30.626 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.626+0000 7f09572be640 10 monclient(hunting): picked mon.noname-a con 0x7f095006c900 addr v2:172.21.15.117:3300/0 2024-05-15T13:20:30.626 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.626+0000 7f09572be640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f09500aeff0 0x7f09500af410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:30.626 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.626+0000 7f09572be640 10 monclient(hunting): picked mon.noname-b con 0x7f09500aeff0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T13:20:30.627 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.626+0000 7f09572be640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0950145e80 addr v1:172.21.15.117:6789/0 2024-05-15T13:20:30.627 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.626+0000 7f09572be640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09500ae0d0 con 0x7f0950145e80 2024-05-15T13:20:30.627 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.626+0000 7f09572be640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:30.627 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.626+0000 7f09572be640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f095006cce0 con 0x7f095006c900 2024-05-15T13:20:30.627 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.626+0000 7f09572be640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:30.627 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.626+0000 7f09572be640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09500b0500 con 0x7f09500aeff0 2024-05-15T13:20:30.627 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.626+0000 7f09572be640 10 monclient(hunting): _renew_subs 2024-05-15T13:20:30.627 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.626+0000 7f09572be640 10 monclient(hunting): authenticate will time out at 3593.868035s 2024-05-15T13:20:30.627 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.626+0000 7f0954859640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f09500aeff0 0x7f09500af410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:30.627 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.627+0000 7f0954859640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f09500aeff0 0x7f09500af410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.117:50374/0 (socket says 172.21.15.117:50374) 2024-05-15T13:20:30.627 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.627+0000 7f0954859640 1 -- 172.21.15.117:0/41243460 learned_addr learned my addr 172.21.15.117:0/41243460 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T13:20:30.628 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.627+0000 7f0954859640 10 monclient(hunting): get_auth_request con 0x7f09500aeff0 auth_method 0 2024-05-15T13:20:30.628 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.627+0000 7f0954859640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:30.628 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.627+0000 7f0954859640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:30.628 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.627+0000 7f0954859640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:30.628 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.627+0000 7f0954859640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:30.628 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.627+0000 7f0954859640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:30.628 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.627+0000 7f0954859640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:30.629 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.629+0000 7f0954859640 10 monclient(hunting): handle_auth_done global_id 18749 payload 274 2024-05-15T13:20:30.629 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.629+0000 7f0954859640 1 -- 172.21.15.117:0/41243460 >> v2:172.21.15.117:3300/0 conn(0x7f095006c900 msgr2=0x7f09500aeab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T13:20:30.629 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.629+0000 7f0954859640 1 --2- 172.21.15.117:0/41243460 >> v2:172.21.15.117:3300/0 conn(0x7f095006c900 0x7f09500aeab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:30.629 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.629+0000 7f0954859640 1 -- 172.21.15.117:0/41243460 >> v1:172.21.15.117:6789/0 conn(0x7f0950145e80 legacy=0x7f0950146330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T13:20:30.629 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.629+0000 7f0954859640 10 monclient: _finish_hunting 0 2024-05-15T13:20:30.629 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.629+0000 7f0954859640 1 monclient: found mon.noname-b 2024-05-15T13:20:30.629 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.629+0000 7f0954859640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:20:30.629 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.629+0000 7f0954859640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.117:3301/0 2024-05-15T13:20:30.629 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.629+0000 7f0954859640 1 -- 172.21.15.117:0/41243460 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09500ae3b0 con 0x7f09500aeff0 2024-05-15T13:20:30.630 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.631+0000 7f0954859640 1 --2- 172.21.15.117:0/41243460 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f09500aeff0 0x7f09500af410 secure :-1 s=READY pgs=3385 cs=0 l=1 rev1=1 crypto rx=0x7f0940009870 tx=0x7f094002b7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4020508d9290f9c7 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:30.633 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.635+0000 7f0947fff640 1 -- 172.21.15.117:0/41243460 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0940038070 con 0x7f09500aeff0 2024-05-15T13:20:30.633 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.635+0000 7f0947fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:30.633 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.635+0000 7f0947fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T13:20:30.633 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.635+0000 7f0947fff640 10 monclient: dump: 2024-05-15T13:20:30.634 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:30.634 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:30.634 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:30.634 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:30.634 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:30.634 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:30.634 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:30.634 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:30.634 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:30.634 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:30.634 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.635+0000 7f0947fff640 10 monclient: _finish_auth 0 2024-05-15T13:20:30.634 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.635+0000 7f0947fff640 10 monclient: _check_auth_tickets 2024-05-15T13:20:30.634 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.635+0000 7f0947fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:20:30.635 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.635+0000 7f0947fff640 1 -- 172.21.15.117:0/41243460 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0940031740 con 0x7f09500aeff0 2024-05-15T13:20:30.635 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.635+0000 7f09572be640 5 monclient: authenticate success, global_id 18749 2024-05-15T13:20:30.635 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.635+0000 7f09572be640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T13:20:30.635 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.635+0000 7f0947fff640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:20:30.635 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.635+0000 7f0947fff640 1 -- 172.21.15.117:0/41243460 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f094002f050 con 0x7f09500aeff0 2024-05-15T13:20:30.635 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.635+0000 7f09572be640 10 monclient: get_monmap_and_config success 2024-05-15T13:20:30.635 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.635+0000 7f09572be640 4 set_mon_vals no callback set 2024-05-15T13:20:30.635 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.635+0000 7f0947fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:30.636 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.635+0000 7f0947fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T13:20:30.636 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.635+0000 7f0947fff640 10 monclient: dump: 2024-05-15T13:20:30.636 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:30.636 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:30.636 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:30.636 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:30.636 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:30.636 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:30.636 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:30.636 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:30.636 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:30.637 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:30.640 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.635+0000 7f09572be640 10 monclient: shutdown 2024-05-15T13:20:30.641 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.635+0000 7f09572be640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:20:30.641 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.635+0000 7f09572be640 1 -- 172.21.15.117:0/41243460 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f09500aeff0 msgr2=0x7f09500af410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:30.641 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.635+0000 7f09572be640 1 --2- 172.21.15.117:0/41243460 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f09500aeff0 0x7f09500af410 secure :-1 s=READY pgs=3385 cs=0 l=1 rev1=1 crypto rx=0x7f0940009870 tx=0x7f094002b7e0 comp rx=0 tx=0).stop 2024-05-15T13:20:30.641 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.635+0000 7f09572be640 1 -- 172.21.15.117:0/41243460 shutdown_connections 2024-05-15T13:20:30.641 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.635+0000 7f09572be640 1 --2- 172.21.15.117:0/41243460 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f09500aeff0 0x7f09500af410 unknown :-1 s=CLOSED pgs=3385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:30.641 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.635+0000 7f09572be640 1 --2- 172.21.15.117:0/41243460 >> v2:172.21.15.117:3300/0 conn(0x7f095006c900 0x7f09500aeab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:30.641 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.635+0000 7f09572be640 1 -- 172.21.15.117:0/41243460 >> 172.21.15.117:0/41243460 conn(0x7f09500b2ac0 msgr2=0x7f09500b2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:20:30.641 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.635+0000 7f09572be640 1 -- 172.21.15.117:0/41243460 shutdown_connections 2024-05-15T13:20:30.641 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.635+0000 7f09572be640 1 -- 172.21.15.117:0/41243460 wait complete. 2024-05-15T13:20:30.641 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.636+0000 7f09572be640 10 monclient: build_initial_monmap 2024-05-15T13:20:30.641 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.636+0000 7f09572be640 10 monclient: monmap: 2024-05-15T13:20:30.641 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T13:20:30.641 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T13:20:30.641 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-05-15T13:20:30.642 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-05-15T13:20:30.642 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T13:20:30.642 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:30.642 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T13:20:30.642 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T13:20:30.642 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T13:20:30.642 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:30.642 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.636+0000 7f09572be640 1 librados: starting msgr at 2024-05-15T13:20:30.642 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.636+0000 7f09572be640 1 librados: starting objecter 2024-05-15T13:20:30.642 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.636+0000 7f09572be640 1 Processor -- start 2024-05-15T13:20:30.642 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.637+0000 7f09572be640 1 -- start start 2024-05-15T13:20:30.642 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.637+0000 7f09572be640 1 librados: setting wanted keys 2024-05-15T13:20:30.643 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.637+0000 7f09572be640 1 librados: calling monclient init 2024-05-15T13:20:30.643 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.637+0000 7f09572be640 10 monclient: init 2024-05-15T13:20:30.643 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.637+0000 7f09572be640 10 monclient: _reopen_session rank -1 2024-05-15T13:20:30.643 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.637+0000 7f09572be640 10 monclient: _add_conns ranks=[0,2,1] 2024-05-15T13:20:30.643 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.637+0000 7f09572be640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f095006c900 0x7f09500b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:30.643 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.637+0000 7f09572be640 10 monclient(hunting): picked mon.noname-a con 0x7f095006c900 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T13:20:30.643 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.637+0000 7f09572be640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f09500aeff0 0x7f09500b67e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:30.643 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.637+0000 7f09572be640 10 monclient(hunting): picked mon.noname-c con 0x7f09500aeff0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T13:20:30.643 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.637+0000 7f09572be640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0950145e80 0x7f09500b39d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:30.643 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.637+0000 7f09572be640 10 monclient(hunting): picked mon.noname-b con 0x7f0950145e80 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T13:20:30.643 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.637+0000 7f09572be640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:30.643 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.637+0000 7f09572be640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0950146a40 con 0x7f095006c900 2024-05-15T13:20:30.643 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.637+0000 7f09572be640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:30.643 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.637+0000 7f09572be640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095006d730 con 0x7f0950145e80 2024-05-15T13:20:30.644 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.637+0000 7f09572be640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:30.644 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.637+0000 7f09572be640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0950146bc0 con 0x7f09500aeff0 2024-05-15T13:20:30.644 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.637+0000 7f09572be640 10 monclient(hunting): _renew_subs 2024-05-15T13:20:30.644 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.637+0000 7f09572be640 10 monclient(hunting): authenticate will time out at 3593.879665s 2024-05-15T13:20:30.644 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.638+0000 7f0954859640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f09500aeff0 0x7f09500b67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:30.644 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.638+0000 7f095585b640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0950145e80 0x7f09500b39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:30.644 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.638+0000 7f0954859640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f09500aeff0 0x7f09500b67e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.117:50376/0 (socket says 172.21.15.117:50376) 2024-05-15T13:20:30.644 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.638+0000 7f0954859640 1 -- 172.21.15.117:0/222718446 learned_addr learned my addr 172.21.15.117:0/222718446 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T13:20:30.644 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.638+0000 7f0954859640 10 monclient(hunting): get_auth_request con 0x7f09500aeff0 auth_method 0 2024-05-15T13:20:30.644 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.638+0000 7f0954859640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:30.644 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.638+0000 7f0954859640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:30.644 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.638+0000 7f0954859640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:30.644 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.638+0000 7f095585b640 10 monclient(hunting): get_auth_request con 0x7f0950145e80 auth_method 0 2024-05-15T13:20:30.644 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.638+0000 7f095585b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:30.644 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.638+0000 7f095585b640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:30.644 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.638+0000 7f095585b640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:30.645 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.639+0000 7f0954859640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:30.645 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.639+0000 7f0954859640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:30.645 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.639+0000 7f0954859640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:30.645 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.639+0000 7f095585b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:30.645 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.639+0000 7f095585b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:30.645 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.639+0000 7f095585b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:30.645 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.640+0000 7f095505a640 1 --2- 172.21.15.117:0/222718446 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f095006c900 0x7f09500b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:30.645 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.640+0000 7f095505a640 10 monclient(hunting): get_auth_request con 0x7f095006c900 auth_method 0 2024-05-15T13:20:30.645 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.640+0000 7f095505a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:30.645 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.640+0000 7f095505a640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:30.645 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.640+0000 7f095505a640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:30.645 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.641+0000 7f095505a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:30.645 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.641+0000 7f095505a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:30.645 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.641+0000 7f095505a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:30.646 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.642+0000 7f095585b640 10 monclient(hunting): handle_auth_done global_id 22168 payload 824 2024-05-15T13:20:30.646 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.642+0000 7f095585b640 1 -- 172.21.15.117:0/222718446 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f09500aeff0 msgr2=0x7f09500b67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:30.646 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.642+0000 7f095585b640 1 --2- 172.21.15.117:0/222718446 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f09500aeff0 0x7f09500b67e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:30.646 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.642+0000 7f095585b640 1 -- 172.21.15.117:0/222718446 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f095006c900 msgr2=0x7f09500b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:30.646 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.642+0000 7f095585b640 1 --2- 172.21.15.117:0/222718446 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f095006c900 0x7f09500b61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:30.646 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.642+0000 7f095585b640 10 monclient: _finish_hunting 0 2024-05-15T13:20:30.646 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.642+0000 7f095585b640 1 monclient: found mon.noname-b 2024-05-15T13:20:30.646 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.642+0000 7f095585b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:20:30.646 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.642+0000 7f095585b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2024-05-15T13:20:30.646 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.642+0000 7f095585b640 1 -- 172.21.15.117:0/222718446 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0950146e90 con 0x7f0950145e80 2024-05-15T13:20:30.648 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.644+0000 7f095585b640 1 --2- 172.21.15.117:0/222718446 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0950145e80 0x7f09500b39d0 secure :-1 s=READY pgs=5449 cs=0 l=1 rev1=1 crypto rx=0x7f094800be20 tx=0x7f0948009ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:30.648 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.644+0000 7f0945ffb640 10 client.?.objecter ms_handle_connect 0x7f0950145e80 2024-05-15T13:20:30.648 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.644+0000 7f0945ffb640 10 client.?.objecter resend_mon_ops 2024-05-15T13:20:30.648 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.647+0000 7f0945ffb640 1 -- 172.21.15.117:0/222718446 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0948019070 con 0x7f0950145e80 2024-05-15T13:20:30.648 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.647+0000 7f0945ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:30.648 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.647+0000 7f0945ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T13:20:30.649 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.648+0000 7f0945ffb640 10 monclient: dump: 2024-05-15T13:20:30.649 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:30.649 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:30.649 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:30.649 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:30.649 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:30.649 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:30.649 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:30.649 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:30.649 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:30.649 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:30.649 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.648+0000 7f0945ffb640 10 monclient: _finish_auth 0 2024-05-15T13:20:30.649 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.648+0000 7f0945ffb640 10 monclient: _check_auth_tickets 2024-05-15T13:20:30.649 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.648+0000 7f0945ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:20:30.649 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.648+0000 7f09572be640 5 monclient: authenticate success, global_id 22168 2024-05-15T13:20:30.649 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.648+0000 7f0945ffb640 1 -- 172.21.15.117:0/222718446 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f094800cdb0 con 0x7f0950145e80 2024-05-15T13:20:30.650 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.648+0000 7f0945ffb640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:20:30.650 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.648+0000 7f0945ffb640 1 -- 172.21.15.117:0/222718446 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0948014680 con 0x7f0950145e80 2024-05-15T13:20:30.650 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.648+0000 7f09572be640 10 monclient: _renew_subs 2024-05-15T13:20:30.650 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.648+0000 7f09572be640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T13:20:30.650 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.648+0000 7f09572be640 1 -- 172.21.15.117:0/222718446 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0950148080 con 0x7f0950145e80 2024-05-15T13:20:30.650 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.648+0000 7f0945ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:30.650 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.648+0000 7f0945ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T13:20:30.650 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.648+0000 7f0945ffb640 10 monclient: dump: 2024-05-15T13:20:30.650 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:30.650 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:30.650 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:30.650 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:30.650 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:30.650 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:30.650 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:30.650 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:30.651 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:30.651 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:30.651 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.648+0000 7f09477fe640 4 set_mon_vals no callback set 2024-05-15T13:20:30.651 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.648+0000 7f09572be640 10 client.22168.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:20:30.651 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.648+0000 7f09572be640 10 monclient: _renew_subs 2024-05-15T13:20:30.651 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.648+0000 7f09572be640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T13:20:30.651 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.648+0000 7f09572be640 1 -- 172.21.15.117:0/222718446 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0950147390 con 0x7f0950145e80 2024-05-15T13:20:30.651 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.648+0000 7f09572be640 1 librados: init done 2024-05-15T13:20:30.651 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.649+0000 7f09572be640 10 client.22168.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:20:30.651 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.649+0000 7f09572be640 15 client.22168.objecter _session_command_op_assign -1 1 2024-05-15T13:20:30.651 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.649+0000 7f09572be640 20 client.22168.objecter _get_session osd=-1 returning homeless 2024-05-15T13:20:30.651 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.649+0000 7f09572be640 10 client.22168.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:20:30.651 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.649+0000 7f09572be640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T13:20:30.651 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.649+0000 7f09572be640 1 -- 172.21.15.117:0/222718446 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f09501475f0 con 0x7f0950145e80 2024-05-15T13:20:30.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.666+0000 7f0945ffb640 1 -- 172.21.15.117:0/222718446 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f094801c070 con 0x7f0950145e80 2024-05-15T13:20:30.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.667+0000 7f0945ffb640 1 --2- 172.21.15.117:0/222718446 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f093406ceb0 0x7f093406f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:30.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.667+0000 7f095505a640 1 --2- 172.21.15.117:0/222718446 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f093406ceb0 0x7f093406f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:30.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.667+0000 7f095505a640 10 monclient: get_auth_request con 0x7f093406ceb0 auth_method 0 2024-05-15T13:20:30.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.667+0000 7f0945ffb640 10 client.22168.objecter ms_handle_connect 0x7f093406ceb0 2024-05-15T13:20:30.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.667+0000 7f095505a640 1 --2- 172.21.15.117:0/222718446 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f093406ceb0 0x7f093406f370 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f094c009990 tx=0x7f094c009110 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:30.668 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.669+0000 7f0945ffb640 1 -- 172.21.15.117:0/222718446 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(1225..1225 src has 912..1225) v4 ==== 6365+0+0 (secure 0 0 0) 0x7f09480570d0 con 0x7f0950145e80 2024-05-15T13:20:30.669 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.669+0000 7f0945ffb640 10 client.22168.objecter ms_dispatch 0x7f0950000b90 osd_map(1225..1225 src has 912..1225) v4 2024-05-15T13:20:30.669 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.669+0000 7f0945ffb640 3 client.22168.objecter handle_osd_map got epochs [1225,1225] > 0 2024-05-15T13:20:30.669 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.669+0000 7f0945ffb640 3 client.22168.objecter handle_osd_map decoding full epoch 1225 2024-05-15T13:20:30.669 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.669+0000 7f0945ffb640 10 client.22168.objecter checking command 1 2024-05-15T13:20:30.669 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.669+0000 7f0945ffb640 1 --2- 172.21.15.117:0/222718446 >> [v2:172.21.15.117:6802/3842021926,v1:172.21.15.117:6803/3842021926] conn(0x7f0934074bc0 0x7f0934077060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:30.669 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.669+0000 7f0945ffb640 20 client.22168.objecter _get_session s=0x7f09340744f0 osd=3 3 2024-05-15T13:20:30.669 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.669+0000 7f0945ffb640 20 client.22168.objecter put_session s=0x7f09340744f0 osd=3 3 2024-05-15T13:20:30.669 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.669+0000 7f0945ffb640 15 client.22168.objecter _session_command_op_remove -1 1 2024-05-15T13:20:30.669 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.669+0000 7f0945ffb640 20 client.22168.objecter _get_session s=0x7f09340744f0 osd=3 3 2024-05-15T13:20:30.669 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.669+0000 7f0945ffb640 20 client.22168.objecter get_session s=0x7f09340744f0 osd=3 3 2024-05-15T13:20:30.669 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.669+0000 7f0945ffb640 15 client.22168.objecter _session_command_op_assign 3 1 2024-05-15T13:20:30.669 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.669+0000 7f0945ffb640 20 client.22168.objecter put_session s=0x7f09340744f0 osd=3 4 2024-05-15T13:20:30.669 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.669+0000 7f0945ffb640 10 client.22168.objecter _send_command 1 2024-05-15T13:20:30.669 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.669+0000 7f0945ffb640 1 -- 172.21.15.117:0/222718446 --> [v2:172.21.15.117:6802/3842021926,v1:172.21.15.117:6803/3842021926] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0934077730 con 0x7f0934074bc0 2024-05-15T13:20:30.670 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.669+0000 7f0945ffb640 20 client.22168.objecter dump_active .. 0 homeless 2024-05-15T13:20:30.670 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.670+0000 7f0954859640 1 --2- 172.21.15.117:0/222718446 >> [v2:172.21.15.117:6802/3842021926,v1:172.21.15.117:6803/3842021926] conn(0x7f0934074bc0 0x7f0934077060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:30.670 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.670+0000 7f0954859640 10 monclient: get_auth_request con 0x7f0934074bc0 auth_method 0 2024-05-15T13:20:30.672 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.673+0000 7f0954859640 1 --2- 172.21.15.117:0/222718446 >> [v2:172.21.15.117:6802/3842021926,v1:172.21.15.117:6803/3842021926] conn(0x7f0934074bc0 0x7f0934077060 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:30.672 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.673+0000 7f0945ffb640 10 client.22168.objecter ms_handle_connect 0x7f0934074bc0 2024-05-15T13:20:30.672 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.673+0000 7f0945ffb640 1 -- 172.21.15.117:0/222718446 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_get_version_reply(handle=1 version=1225) v2 ==== 24+0+0 (secure 0 0 0) 0x7f09480552d0 con 0x7f0950145e80 2024-05-15T13:20:30.672 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.673+0000 7f0945ffb640 10 monclient: handle_get_version_reply finishing 1 version 1225 2024-05-15T13:20:30.697 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.698+0000 7f0945ffb640 1 -- 172.21.15.117:0/222718446 <== osd.3 v2:172.21.15.117:6802/3842021926 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f0934077730 con 0x7f0934074bc0 2024-05-15T13:20:30.697 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.698+0000 7f0945ffb640 10 client.22168.objecter ms_dispatch 0x7f0950000b90 command_reply(tid 1: 0 ) v1 2024-05-15T13:20:30.697 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.698+0000 7f0945ffb640 10 client.22168.objecter _finish_command 1 = system:0 2024-05-15T13:20:30.697 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.698+0000 7f0945ffb640 20 client.22168.objecter put_session s=0x7f09340744f0 osd=3 4 2024-05-15T13:20:30.697 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.698+0000 7f0945ffb640 15 client.22168.objecter _session_command_op_remove 3 1 2024-05-15T13:20:30.710 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.711+0000 7f09572be640 10 client.22168.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-05-15T13:20:30.710 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.711+0000 7f09572be640 15 client.22168.objecter _session_command_op_assign -1 2 2024-05-15T13:20:30.710 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.711+0000 7f09572be640 20 client.22168.objecter _get_session s=0x7f09340744f0 osd=3 3 2024-05-15T13:20:30.710 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.711+0000 7f09572be640 20 client.22168.objecter put_session s=0x7f09340744f0 osd=3 3 2024-05-15T13:20:30.710 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.711+0000 7f09572be640 20 client.22168.objecter _get_session s=0x7f09340744f0 osd=3 3 2024-05-15T13:20:30.710 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.711+0000 7f09572be640 15 client.22168.objecter _session_command_op_remove -1 2 2024-05-15T13:20:30.710 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.711+0000 7f09572be640 20 client.22168.objecter get_session s=0x7f09340744f0 osd=3 3 2024-05-15T13:20:30.710 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.711+0000 7f09572be640 15 client.22168.objecter _session_command_op_assign 3 2 2024-05-15T13:20:30.710 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.711+0000 7f09572be640 20 client.22168.objecter put_session s=0x7f09340744f0 osd=3 4 2024-05-15T13:20:30.711 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.711+0000 7f09572be640 10 client.22168.objecter _send_command 2 2024-05-15T13:20:30.711 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.711+0000 7f09572be640 1 -- 172.21.15.117:0/222718446 --> [v2:172.21.15.117:6802/3842021926,v1:172.21.15.117:6803/3842021926] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f095004f0e0 con 0x7f0934074bc0 2024-05-15T13:20:30.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.724+0000 7f0945ffb640 1 -- 172.21.15.117:0/222718446 <== osd.3 v2:172.21.15.117:6802/3842021926 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f095004f0e0 con 0x7f0934074bc0 2024-05-15T13:20:30.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.724+0000 7f0945ffb640 10 client.22168.objecter ms_dispatch 0x7f0950000b90 command_reply(tid 2: 0 ) v1 2024-05-15T13:20:30.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.724+0000 7f0945ffb640 10 client.22168.objecter _finish_command 2 = system:0 2024-05-15T13:20:30.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.724+0000 7f0945ffb640 20 client.22168.objecter put_session s=0x7f09340744f0 osd=3 4 2024-05-15T13:20:30.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.724+0000 7f0945ffb640 15 client.22168.objecter _session_command_op_remove 3 2 2024-05-15T13:20:30.724 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-05-15T13:20:30.724 INFO:teuthology.orchestra.run.smithi117.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-15T13:20:30.724 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-05-15T13:20:30.726 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.724+0000 7f09572be640 10 librados: watch_flush enter 2024-05-15T13:20:30.726 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.724+0000 7f09572be640 10 librados: watch_flush exit 2024-05-15T13:20:30.726 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.724+0000 7f09572be640 10 client.22168.objecter close_session for osd.3 2024-05-15T13:20:30.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.724+0000 7f09572be640 1 -- 172.21.15.117:0/222718446 >> [v2:172.21.15.117:6802/3842021926,v1:172.21.15.117:6803/3842021926] conn(0x7f0934074bc0 msgr2=0x7f0934077060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:30.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.724+0000 7f09572be640 1 --2- 172.21.15.117:0/222718446 >> [v2:172.21.15.117:6802/3842021926,v1:172.21.15.117:6803/3842021926] conn(0x7f0934074bc0 0x7f0934077060 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:30.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.724+0000 7f09572be640 20 client.22168.objecter put_session s=0x7f09340744f0 osd=3 1 2024-05-15T13:20:30.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.724+0000 7f09572be640 20 client.22168.objecter shutdown clearing up homeless session... 2024-05-15T13:20:30.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.724+0000 7f09572be640 10 client.22168.objecter successfully canceled tick 2024-05-15T13:20:30.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.724+0000 7f09572be640 1 -- 172.21.15.117:0/222718446 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f093406ceb0 msgr2=0x7f093406f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:30.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.724+0000 7f09572be640 1 --2- 172.21.15.117:0/222718446 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f093406ceb0 0x7f093406f370 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f094c009990 tx=0x7f094c009110 comp rx=0 tx=0).stop 2024-05-15T13:20:30.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.724+0000 7f09572be640 10 monclient: shutdown 2024-05-15T13:20:30.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.724+0000 7f09572be640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:20:30.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.724+0000 7f09572be640 1 -- 172.21.15.117:0/222718446 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0950145e80 msgr2=0x7f09500b39d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:30.729 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.724+0000 7f09572be640 1 --2- 172.21.15.117:0/222718446 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0950145e80 0x7f09500b39d0 secure :-1 s=READY pgs=5449 cs=0 l=1 rev1=1 crypto rx=0x7f094800be20 tx=0x7f0948009ef0 comp rx=0 tx=0).stop 2024-05-15T13:20:30.729 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.724+0000 7f095585b640 1 -- 172.21.15.117:0/222718446 reap_dead start 2024-05-15T13:20:30.729 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.724+0000 7f09572be640 1 -- 172.21.15.117:0/222718446 shutdown_connections 2024-05-15T13:20:30.729 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.724+0000 7f09572be640 1 -- 172.21.15.117:0/222718446 >> 172.21.15.117:0/222718446 conn(0x7f09500b2ac0 msgr2=0x7f0950143750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:20:30.729 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.725+0000 7f09572be640 1 -- 172.21.15.117:0/222718446 shutdown_connections 2024-05-15T13:20:30.729 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.725+0000 7f09572be640 1 -- 172.21.15.117:0/222718446 wait complete. 2024-05-15T13:20:30.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.725+0000 7f09572be640 1 librados: shutdown 2024-05-15T13:20:30.734 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-05-15T13:20:30.836 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.837+0000 7f3e19c7e640 10 monclient: get_monmap_and_config 2024-05-15T13:20:30.836 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.837+0000 7f3e19c7e640 10 monclient: build_initial_monmap 2024-05-15T13:20:30.836 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.837+0000 7f3e19c7e640 10 monclient: monmap: 2024-05-15T13:20:30.837 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T13:20:30.837 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:30.837 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T13:20:30.838873+0000 2024-05-15T13:20:30.837 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T13:20:30.838873+0000 2024-05-15T13:20:30.837 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T13:20:30.837 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:30.837 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T13:20:30.837 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T13:20:30.837 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T13:20:30.837 INFO:teuthology.orchestra.run.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T13:20:30.837 INFO:teuthology.orchestra.run.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T13:20:30.837 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:30.839 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.838+0000 7f3e19c7e640 1 Processor -- start 2024-05-15T13:20:30.839 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.838+0000 7f3e19c7e640 1 -- start start 2024-05-15T13:20:30.839 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.838+0000 7f3e19c7e640 10 monclient: init 2024-05-15T13:20:30.839 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.839+0000 7f3e19c7e640 10 monclient: _reopen_session rank -1 2024-05-15T13:20:30.839 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.839+0000 7f3e19c7e640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-05-15T13:20:30.839 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.839+0000 7f3e19c7e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3e140b4d00 addr v1:172.21.15.117:6789/0 2024-05-15T13:20:30.839 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.839+0000 7f3e19c7e640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f3e140b50e0 0x7f3e140b1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:30.839 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.839+0000 7f3e19c7e640 10 monclient(hunting): picked mon.noname-a con 0x7f3e140b50e0 addr v2:172.21.15.117:3300/0 2024-05-15T13:20:30.839 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.839+0000 7f3e19c7e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3e140a94b0 addr v1:172.21.15.178:6789/0 2024-05-15T13:20:30.839 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.839+0000 7f3e19c7e640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e140ab1b0 con 0x7f3e140b4d00 2024-05-15T13:20:30.839 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.839+0000 7f3e19c7e640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e140a9960 con 0x7f3e140a94b0 2024-05-15T13:20:30.839 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.839+0000 7f3e19c7e640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:30.839 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.839+0000 7f3e19c7e640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e140b23c0 con 0x7f3e140b50e0 2024-05-15T13:20:30.839 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.839+0000 7f3e19c7e640 10 monclient(hunting): _renew_subs 2024-05-15T13:20:30.840 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.839+0000 7f3e19c7e640 10 monclient(hunting): authenticate will time out at 3594.081282s 2024-05-15T13:20:30.840 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.840+0000 7f3e13fff640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f3e140a94b0 0x7f3e140b0fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.117:36826/0 (socket says 172.21.15.117:36826) 2024-05-15T13:20:30.840 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.840+0000 7f3e13fff640 1 -- 172.21.15.117:0/3977007576 learned_addr learned my addr 172.21.15.117:0/3977007576 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-05-15T13:20:30.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.889+0000 7f3e0bfff640 1 --2- 172.21.15.117:0/3977007576 >> v2:172.21.15.117:3300/0 conn(0x7f3e140b50e0 0x7f3e140b1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:30.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.889+0000 7f3e0bfff640 10 monclient(hunting): get_auth_request con 0x7f3e140b50e0 auth_method 0 2024-05-15T13:20:30.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.889+0000 7f3e0bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:30.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.889+0000 7f3e0bfff640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:30.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.889+0000 7f3e0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:30.889 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.890+0000 7f3e0bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:30.889 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.890+0000 7f3e0bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:30.889 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.890+0000 7f3e0bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:30.891 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.892+0000 7f3e0bfff640 10 monclient(hunting): handle_auth_done global_id 16047 payload 274 2024-05-15T13:20:30.891 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.892+0000 7f3e0bfff640 1 -- 172.21.15.117:0/3977007576 >> v1:172.21.15.178:6789/0 conn(0x7f3e140a94b0 legacy=0x7f3e140b0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:30.891 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.892+0000 7f3e0bfff640 1 -- 172.21.15.117:0/3977007576 >> v1:172.21.15.117:6789/0 conn(0x7f3e140b4d00 legacy=0x7f3e14145c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:30.891 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.892+0000 7f3e0bfff640 10 monclient: _finish_hunting 0 2024-05-15T13:20:30.891 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.892+0000 7f3e0bfff640 1 monclient: found mon.noname-a 2024-05-15T13:20:30.891 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.892+0000 7f3e0bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:20:30.891 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.892+0000 7f3e0bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.117:3300/0 2024-05-15T13:20:30.891 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.892+0000 7f3e0bfff640 1 -- 172.21.15.117:0/3977007576 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e140a9b30 con 0x7f3e140b50e0 2024-05-15T13:20:30.895 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.896+0000 7f3e0bfff640 1 --2- 172.21.15.117:0/3977007576 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3e140b50e0 0x7f3e140b1e80 secure :-1 s=READY pgs=3299 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc0097a0 tx=0x7f3dfc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0e8ade21440a22a server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:30.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.899+0000 7f3e12ffd640 1 -- 172.21.15.117:0/3977007576 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dfc035020 con 0x7f3e140b50e0 2024-05-15T13:20:30.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.899+0000 7f3e12ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:30.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.899+0000 7f3e12ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T13:20:30.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.899+0000 7f3e12ffd640 10 monclient: dump: 2024-05-15T13:20:30.898 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:30.898 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:30.898 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:30.898 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:30.898 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:30.898 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:30.898 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:30.898 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:30.898 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:30.898 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:30.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.899+0000 7f3e12ffd640 10 monclient: _finish_auth 0 2024-05-15T13:20:30.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.899+0000 7f3e12ffd640 10 monclient: _check_auth_tickets 2024-05-15T13:20:30.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.899+0000 7f3e12ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:20:30.899 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.899+0000 7f3e19c7e640 5 monclient: authenticate success, global_id 16047 2024-05-15T13:20:30.899 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.899+0000 7f3e19c7e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T13:20:30.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.900+0000 7f3e12ffd640 1 -- 172.21.15.117:0/3977007576 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dfc031830 con 0x7f3e140b50e0 2024-05-15T13:20:30.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.900+0000 7f3e12ffd640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:20:30.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.900+0000 7f3e12ffd640 1 -- 172.21.15.117:0/3977007576 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dfc031c00 con 0x7f3e140b50e0 2024-05-15T13:20:30.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.900+0000 7f3e12ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:30.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.900+0000 7f3e12ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T13:20:30.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.900+0000 7f3e12ffd640 10 monclient: dump: 2024-05-15T13:20:30.900 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:30.900 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:30.900 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:30.900 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:30.900 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:30.900 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:30.901 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:30.901 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:30.901 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:30.901 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:30.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.900+0000 7f3e19c7e640 10 monclient: get_monmap_and_config success 2024-05-15T13:20:30.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.900+0000 7f3e19c7e640 4 set_mon_vals no callback set 2024-05-15T13:20:30.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.901+0000 7f3e19c7e640 10 monclient: shutdown 2024-05-15T13:20:30.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.901+0000 7f3e19c7e640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:20:30.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.901+0000 7f3e19c7e640 1 -- 172.21.15.117:0/3977007576 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3e140b50e0 msgr2=0x7f3e140b1e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:30.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.901+0000 7f3e19c7e640 1 --2- 172.21.15.117:0/3977007576 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3e140b50e0 0x7f3e140b1e80 secure :-1 s=READY pgs=3299 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc0097a0 tx=0x7f3dfc02b8d0 comp rx=0 tx=0).stop 2024-05-15T13:20:30.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.901+0000 7f3e19c7e640 1 -- 172.21.15.117:0/3977007576 shutdown_connections 2024-05-15T13:20:30.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.901+0000 7f3e19c7e640 1 --2- 172.21.15.117:0/3977007576 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3e140b50e0 0x7f3e140b1e80 unknown :-1 s=CLOSED pgs=3299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:30.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.901+0000 7f3e19c7e640 1 -- 172.21.15.117:0/3977007576 >> 172.21.15.117:0/3977007576 conn(0x7f3e14066cc0 msgr2=0x7f3e140670d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:20:30.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.901+0000 7f3e19c7e640 1 -- 172.21.15.117:0/3977007576 shutdown_connections 2024-05-15T13:20:30.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.901+0000 7f3e19c7e640 1 -- 172.21.15.117:0/3977007576 wait complete. 2024-05-15T13:20:30.904 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.902+0000 7f3e19c7e640 10 monclient: build_initial_monmap 2024-05-15T13:20:30.904 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.902+0000 7f3e19c7e640 10 monclient: monmap: 2024-05-15T13:20:30.904 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T13:20:30.904 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T13:20:30.904 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-05-15T13:20:30.904 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-05-15T13:20:30.904 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T13:20:30.904 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:30.904 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T13:20:30.905 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T13:20:30.905 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T13:20:30.905 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:30.905 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.902+0000 7f3e19c7e640 1 librados: starting msgr at 2024-05-15T13:20:30.905 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.902+0000 7f3e19c7e640 1 librados: starting objecter 2024-05-15T13:20:30.905 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.902+0000 7f3e19c7e640 1 Processor -- start 2024-05-15T13:20:30.905 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.902+0000 7f3e19c7e640 1 -- start start 2024-05-15T13:20:30.905 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.902+0000 7f3e19c7e640 1 librados: setting wanted keys 2024-05-15T13:20:30.905 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.902+0000 7f3e19c7e640 1 librados: calling monclient init 2024-05-15T13:20:30.905 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.902+0000 7f3e19c7e640 10 monclient: init 2024-05-15T13:20:30.905 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.903+0000 7f3e19c7e640 10 monclient: _reopen_session rank -1 2024-05-15T13:20:30.905 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.903+0000 7f3e19c7e640 10 monclient: _add_conns ranks=[2,0,1] 2024-05-15T13:20:30.905 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.903+0000 7f3e19c7e640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3e140a94b0 0x7f3e141db750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:30.905 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.903+0000 7f3e19c7e640 10 monclient(hunting): picked mon.noname-c con 0x7f3e140a94b0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T13:20:30.905 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.903+0000 7f3e19c7e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3e140b4d00 0x7f3e1413d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:30.906 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.903+0000 7f3e19c7e640 10 monclient(hunting): picked mon.noname-a con 0x7f3e140b4d00 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T13:20:30.906 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.903+0000 7f3e19c7e640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3e1413d730 0x7f3e14135d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:30.906 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.903+0000 7f3e19c7e640 10 monclient(hunting): picked mon.noname-b con 0x7f3e1413d730 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T13:20:30.906 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.903+0000 7f3e19c7e640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:30.906 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.903+0000 7f3e19c7e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e1414e1d0 con 0x7f3e140b4d00 2024-05-15T13:20:30.906 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.903+0000 7f3e19c7e640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:30.906 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.903+0000 7f3e19c7e640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e140b16f0 con 0x7f3e1413d730 2024-05-15T13:20:30.906 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.903+0000 7f3e19c7e640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:30.906 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.903+0000 7f3e19c7e640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e1414e350 con 0x7f3e140a94b0 2024-05-15T13:20:30.906 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.903+0000 7f3e19c7e640 10 monclient(hunting): _renew_subs 2024-05-15T13:20:30.906 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.903+0000 7f3e19c7e640 10 monclient(hunting): authenticate will time out at 3594.145056s 2024-05-15T13:20:30.906 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.903+0000 7f3e0bfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3e140b4d00 0x7f3e1413d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:30.906 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.904+0000 7f3e0bfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3e140b4d00 0x7f3e1413d1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:44254/0 (socket says 172.21.15.117:44254) 2024-05-15T13:20:30.906 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.904+0000 7f3e0bfff640 1 -- 172.21.15.117:0/2844035972 learned_addr learned my addr 172.21.15.117:0/2844035972 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T13:20:30.907 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.904+0000 7f3e0bfff640 10 monclient(hunting): get_auth_request con 0x7f3e140b4d00 auth_method 0 2024-05-15T13:20:30.907 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.904+0000 7f3e0bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:30.907 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.904+0000 7f3e0bfff640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:30.907 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.904+0000 7f3e0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:30.907 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.904+0000 7f3e0bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:30.907 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.904+0000 7f3e0bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:30.907 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.904+0000 7f3e0bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:30.907 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.908+0000 7f3e0bfff640 10 monclient(hunting): handle_auth_done global_id 16053 payload 824 2024-05-15T13:20:30.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.908+0000 7f3e0bfff640 1 -- 172.21.15.117:0/2844035972 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3e140a94b0 msgr2=0x7f3e141db750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:30.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.908+0000 7f3e0bfff640 1 --2- 172.21.15.117:0/2844035972 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3e140a94b0 0x7f3e141db750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:30.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.908+0000 7f3e0bfff640 1 -- 172.21.15.117:0/2844035972 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3e1413d730 msgr2=0x7f3e14135d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:30.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.908+0000 7f3e0bfff640 1 --2- 172.21.15.117:0/2844035972 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3e1413d730 0x7f3e14135d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:30.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.908+0000 7f3e0bfff640 10 monclient: _finish_hunting 0 2024-05-15T13:20:30.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.908+0000 7f3e0bfff640 1 monclient: found mon.noname-a 2024-05-15T13:20:30.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.908+0000 7f3e0bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:20:30.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.909+0000 7f3e0bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.117:3300/0 2024-05-15T13:20:30.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.909+0000 7f3e0bfff640 1 -- 172.21.15.117:0/2844035972 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e1413db80 con 0x7f3e140b4d00 2024-05-15T13:20:30.910 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.912+0000 7f3e0bfff640 1 --2- 172.21.15.117:0/2844035972 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3e140b4d00 0x7f3e1413d1f0 secure :-1 s=READY pgs=3300 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc034040 tx=0x7f3dfc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:30.910 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.912+0000 7f3e10ff9640 10 client.?.objecter ms_handle_connect 0x7f3e140b4d00 2024-05-15T13:20:30.910 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.912+0000 7f3e10ff9640 10 client.?.objecter resend_mon_ops 2024-05-15T13:20:30.915 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.916+0000 7f3e10ff9640 1 -- 172.21.15.117:0/2844035972 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dfc040070 con 0x7f3e140b4d00 2024-05-15T13:20:30.915 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.916+0000 7f3e10ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:30.915 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.916+0000 7f3e10ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T13:20:30.915 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.916+0000 7f3e10ff9640 10 monclient: dump: 2024-05-15T13:20:30.915 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:30.915 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:30.915 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:30.916 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:30.916 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:30.916 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:30.916 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:30.916 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:30.916 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:30.916 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:30.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.916+0000 7f3e10ff9640 10 monclient: _finish_auth 0 2024-05-15T13:20:30.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.916+0000 7f3e10ff9640 10 monclient: _check_auth_tickets 2024-05-15T13:20:30.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.916+0000 7f3e10ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:20:30.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.916+0000 7f3e10ff9640 1 -- 172.21.15.117:0/2844035972 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dfc039a20 con 0x7f3e140b4d00 2024-05-15T13:20:30.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.916+0000 7f3e19c7e640 5 monclient: authenticate success, global_id 16053 2024-05-15T13:20:30.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.916+0000 7f3e19c7e640 10 monclient: _renew_subs 2024-05-15T13:20:30.917 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.916+0000 7f3e19c7e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T13:20:30.917 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.916+0000 7f3e19c7e640 1 -- 172.21.15.117:0/2844035972 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e140ae2e0 con 0x7f3e140b4d00 2024-05-15T13:20:30.917 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.916+0000 7f3e10ff9640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:20:30.917 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.916+0000 7f3e10ff9640 1 -- 172.21.15.117:0/2844035972 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dfc035040 con 0x7f3e140b4d00 2024-05-15T13:20:30.917 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.916+0000 7f3e10ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:30.917 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.916+0000 7f3e10ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T13:20:30.917 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.916+0000 7f3e10ff9640 10 monclient: dump: 2024-05-15T13:20:30.917 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:30.917 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:30.917 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:30.917 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:30.917 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:30.917 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:30.917 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:30.917 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:30.917 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:30.918 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:30.918 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.916+0000 7f3e19c7e640 10 client.16053.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:20:30.918 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.916+0000 7f3e19c7e640 10 monclient: _renew_subs 2024-05-15T13:20:30.918 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.916+0000 7f3e19c7e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T13:20:30.918 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.916+0000 7f3e19c7e640 1 -- 172.21.15.117:0/2844035972 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e141dc270 con 0x7f3e140b4d00 2024-05-15T13:20:30.918 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.916+0000 7f3e19c7e640 1 librados: init done 2024-05-15T13:20:30.918 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.916+0000 7f3e127fc640 4 set_mon_vals no callback set 2024-05-15T13:20:30.919 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.917+0000 7f3e19c7e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T13:20:30.919 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.917+0000 7f3e19c7e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:20:30.919 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.917+0000 7f3e19c7e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T13:20:30.919 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.917+0000 7f3e19c7e640 1 -- 172.21.15.117:0/2844035972 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e1404f0e0 con 0x7f3e140b4d00 2024-05-15T13:20:30.940 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.941+0000 7f3e10ff9640 1 -- 172.21.15.117:0/2844035972 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f3dfc038020 con 0x7f3e140b4d00 2024-05-15T13:20:30.940 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.942+0000 7f3e10ff9640 1 --2- 172.21.15.117:0/2844035972 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f3df406ce60 0x7f3df406f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:30.941 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.942+0000 7f3e137fe640 1 --2- 172.21.15.117:0/2844035972 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f3df406ce60 0x7f3df406f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:30.941 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.942+0000 7f3e137fe640 10 monclient: get_auth_request con 0x7f3df406ce60 auth_method 0 2024-05-15T13:20:30.941 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.942+0000 7f3e10ff9640 10 client.16053.objecter ms_handle_connect 0x7f3df406ce60 2024-05-15T13:20:30.941 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.942+0000 7f3e137fe640 1 --2- 172.21.15.117:0/2844035972 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f3df406ce60 0x7f3df406f320 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f3e04000f60 tx=0x7f3e040079d0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:30.943 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.944+0000 7f3e10ff9640 1 -- 172.21.15.117:0/2844035972 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(1225..1225 src has 912..1225) v4 ==== 6365+0+0 (secure 0 0 0) 0x7f3dfc04be20 con 0x7f3e140b4d00 2024-05-15T13:20:30.943 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.944+0000 7f3e10ff9640 10 client.16053.objecter ms_dispatch 0x7f3e14000b90 osd_map(1225..1225 src has 912..1225) v4 2024-05-15T13:20:30.943 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.944+0000 7f3e10ff9640 3 client.16053.objecter handle_osd_map got epochs [1225,1225] > 0 2024-05-15T13:20:30.943 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.944+0000 7f3e10ff9640 3 client.16053.objecter handle_osd_map decoding full epoch 1225 2024-05-15T13:20:30.943 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:30.944+0000 7f3e10ff9640 20 client.16053.objecter dump_active .. 0 homeless 2024-05-15T13:20:31.187 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.188+0000 7f3e10ff9640 1 -- 172.21.15.117:0/2844035972 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f3dfc07af90 con 0x7f3e140b4d00 2024-05-15T13:20:31.187 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.188+0000 7f3e10ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:20:31.187 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.188+0000 7f3e10ff9640 10 monclient: _finish_command 1 = system:0 2024-05-15T13:20:31.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.369+0000 7f3e19c7e640 1 -- 172.21.15.117:0/2844035972 --> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f3e140b75e0 con 0x7f3df406ce60 2024-05-15T13:20:31.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.369+0000 7f3e10ff9640 1 -- 172.21.15.117:0/2844035972 <== mgr.4107 v2:172.21.15.117:6800/1785766523 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f3e140b75e0 con 0x7f3df406ce60 2024-05-15T13:20:31.371 INFO:teuthology.orchestra.run.smithi117.stderr:instructed osd(s) 3 to deep-scrub 2024-05-15T13:20:31.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.371+0000 7f3e19c7e640 10 librados: watch_flush enter 2024-05-15T13:20:31.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.371+0000 7f3e19c7e640 10 librados: watch_flush exit 2024-05-15T13:20:31.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.371+0000 7f3e19c7e640 20 client.16053.objecter shutdown clearing up homeless session... 2024-05-15T13:20:31.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.371+0000 7f3e19c7e640 10 client.16053.objecter successfully canceled tick 2024-05-15T13:20:31.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.371+0000 7f3e19c7e640 1 -- 172.21.15.117:0/2844035972 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f3df406ce60 msgr2=0x7f3df406f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:31.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.371+0000 7f3e19c7e640 1 --2- 172.21.15.117:0/2844035972 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f3df406ce60 0x7f3df406f320 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f3e04000f60 tx=0x7f3e040079d0 comp rx=0 tx=0).stop 2024-05-15T13:20:31.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.371+0000 7f3e19c7e640 10 monclient: shutdown 2024-05-15T13:20:31.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.371+0000 7f3e19c7e640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:20:31.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.371+0000 7f3e19c7e640 1 -- 172.21.15.117:0/2844035972 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3e140b4d00 msgr2=0x7f3e1413d1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:31.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.371+0000 7f3e19c7e640 1 --2- 172.21.15.117:0/2844035972 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3e140b4d00 0x7f3e1413d1f0 secure :-1 s=READY pgs=3300 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc034040 tx=0x7f3dfc0046c0 comp rx=0 tx=0).stop 2024-05-15T13:20:31.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.371+0000 7f3e19c7e640 1 -- 172.21.15.117:0/2844035972 shutdown_connections 2024-05-15T13:20:31.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.371+0000 7f3e19c7e640 1 --2- 172.21.15.117:0/2844035972 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f3df406ce60 0x7f3df406f320 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:31.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.371+0000 7f3e19c7e640 1 --2- 172.21.15.117:0/2844035972 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3e1413d730 0x7f3e14135d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:31.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.371+0000 7f3e19c7e640 1 --2- 172.21.15.117:0/2844035972 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3e140b4d00 0x7f3e1413d1f0 unknown :-1 s=CLOSED pgs=3300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:31.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.371+0000 7f3e19c7e640 1 --2- 172.21.15.117:0/2844035972 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3e140a94b0 0x7f3e141db750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:31.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.371+0000 7f3e19c7e640 1 -- 172.21.15.117:0/2844035972 >> 172.21.15.117:0/2844035972 conn(0x7f3e14066cc0 msgr2=0x7f3e14142610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:20:31.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.371+0000 7f3e19c7e640 1 -- 172.21.15.117:0/2844035972 shutdown_connections 2024-05-15T13:20:31.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.371+0000 7f3e19c7e640 1 -- 172.21.15.117:0/2844035972 wait complete. 2024-05-15T13:20:31.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.371+0000 7f3e19c7e640 1 librados: shutdown 2024-05-15T13:20:31.382 INFO:tasks.ceph:Scrubbing osd.4 2024-05-15T13:20:31.383 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-05-15T13:20:31.487 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.488+0000 7ff4084ca640 10 monclient: get_monmap_and_config 2024-05-15T13:20:31.487 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.488+0000 7ff4084ca640 10 monclient: build_initial_monmap 2024-05-15T13:20:31.487 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.488+0000 7ff4084ca640 10 monclient: monmap: 2024-05-15T13:20:31.487 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T13:20:31.488 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:31.488 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T13:20:31.489995+0000 2024-05-15T13:20:31.488 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T13:20:31.489995+0000 2024-05-15T13:20:31.488 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T13:20:31.488 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:31.488 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T13:20:31.488 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T13:20:31.488 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T13:20:31.488 INFO:teuthology.orchestra.run.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T13:20:31.488 INFO:teuthology.orchestra.run.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T13:20:31.488 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:31.490 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.489+0000 7ff4084ca640 1 Processor -- start 2024-05-15T13:20:31.490 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.489+0000 7ff4084ca640 1 -- start start 2024-05-15T13:20:31.490 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.489+0000 7ff4084ca640 10 monclient: init 2024-05-15T13:20:31.490 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.490+0000 7ff4084ca640 10 monclient: _reopen_session rank -1 2024-05-15T13:20:31.490 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.490+0000 7ff4084ca640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-05-15T13:20:31.490 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.490+0000 7ff4084ca640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7ff40006c900 0x7ff4000aeab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:31.490 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.490+0000 7ff4084ca640 10 monclient(hunting): picked mon.noname-c con 0x7ff40006c900 addr v2:172.21.15.178:3300/0 2024-05-15T13:20:31.490 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.490+0000 7ff4084ca640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff4000aeff0 addr v1:172.21.15.178:6789/0 2024-05-15T13:20:31.490 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.490+0000 7ff4084ca640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7ff4000af440 0x7ff400146590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:31.490 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.490+0000 7ff4084ca640 10 monclient(hunting): picked mon.noname-a con 0x7ff4000af440 addr v2:172.21.15.117:3300/0 2024-05-15T13:20:31.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.490+0000 7ff4084ca640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4000ae0d0 con 0x7ff4000aeff0 2024-05-15T13:20:31.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.490+0000 7ff4084ca640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:31.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.490+0000 7ff4084ca640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff40006cce0 con 0x7ff4000af440 2024-05-15T13:20:31.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.490+0000 7ff4084ca640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:31.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.490+0000 7ff4084ca640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4000b0500 con 0x7ff40006c900 2024-05-15T13:20:31.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.490+0000 7ff4084ca640 10 monclient(hunting): _renew_subs 2024-05-15T13:20:31.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.490+0000 7ff4084ca640 10 monclient(hunting): authenticate will time out at 3594.732258s 2024-05-15T13:20:31.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.490+0000 7ff406a67640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7ff4000af440 0x7ff400146590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:31.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.491+0000 7ff406a67640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7ff4000af440 0x7ff400146590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:44258/0 (socket says 172.21.15.117:44258) 2024-05-15T13:20:31.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.491+0000 7ff406a67640 1 -- 172.21.15.117:0/2783724844 learned_addr learned my addr 172.21.15.117:0/2783724844 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T13:20:31.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.491+0000 7ff406a67640 10 monclient(hunting): get_auth_request con 0x7ff4000af440 auth_method 0 2024-05-15T13:20:31.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.491+0000 7ff406a67640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:31.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.491+0000 7ff406a67640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:31.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.491+0000 7ff406a67640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:31.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.491+0000 7ff406266640 1 --2- 172.21.15.117:0/2783724844 >> v2:172.21.15.178:3300/0 conn(0x7ff40006c900 0x7ff4000aeab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:31.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.491+0000 7ff406a67640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:31.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.491+0000 7ff406a67640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:31.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.491+0000 7ff406a67640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:31.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.492+0000 7ff406266640 10 monclient(hunting): get_auth_request con 0x7ff40006c900 auth_method 0 2024-05-15T13:20:31.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.492+0000 7ff406266640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:31.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.492+0000 7ff406266640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:31.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.492+0000 7ff406266640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:31.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.492+0000 7ff406266640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:31.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.492+0000 7ff406266640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:31.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.492+0000 7ff406266640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:31.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.493+0000 7ff406a67640 10 monclient(hunting): handle_auth_done global_id 16056 payload 274 2024-05-15T13:20:31.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.493+0000 7ff406a67640 1 -- 172.21.15.117:0/2783724844 >> v2:172.21.15.178:3300/0 conn(0x7ff40006c900 msgr2=0x7ff4000aeab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T13:20:31.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.493+0000 7ff406a67640 1 --2- 172.21.15.117:0/2783724844 >> v2:172.21.15.178:3300/0 conn(0x7ff40006c900 0x7ff4000aeab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:31.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.493+0000 7ff406a67640 1 -- 172.21.15.117:0/2783724844 >> v1:172.21.15.178:6789/0 conn(0x7ff4000aeff0 legacy=0x7ff400145e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:31.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.493+0000 7ff406a67640 10 monclient: _finish_hunting 0 2024-05-15T13:20:31.494 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.493+0000 7ff406a67640 1 monclient: found mon.noname-a 2024-05-15T13:20:31.494 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.493+0000 7ff406a67640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:20:31.494 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.493+0000 7ff406a67640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.117:3300/0 2024-05-15T13:20:31.494 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.493+0000 7ff406a67640 1 -- 172.21.15.117:0/2783724844 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4000ae3b0 con 0x7ff4000af440 2024-05-15T13:20:31.494 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.495+0000 7ff406a67640 1 --2- 172.21.15.117:0/2783724844 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff4000af440 0x7ff400146590 secure :-1 s=READY pgs=3301 cs=0 l=1 rev1=1 crypto rx=0x7ff400072c00 tx=0x7ff3f802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69a8e5ccf86c08e2 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:31.497 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.498+0000 7ff405264640 1 -- 172.21.15.117:0/2783724844 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f802ca80 con 0x7ff4000af440 2024-05-15T13:20:31.497 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.498+0000 7ff405264640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:31.497 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.498+0000 7ff405264640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T13:20:31.497 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.498+0000 7ff405264640 10 monclient: dump: 2024-05-15T13:20:31.497 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:31.497 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:31.498 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:31.498 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:31.498 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:31.498 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:31.498 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:31.498 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:31.498 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:31.498 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:31.498 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.498+0000 7ff405264640 10 monclient: _finish_auth 0 2024-05-15T13:20:31.498 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.498+0000 7ff405264640 10 monclient: _check_auth_tickets 2024-05-15T13:20:31.498 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.498+0000 7ff405264640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:20:31.498 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.498+0000 7ff405264640 1 -- 172.21.15.117:0/2783724844 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3f802cc20 con 0x7ff4000af440 2024-05-15T13:20:31.498 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.498+0000 7ff4084ca640 5 monclient: authenticate success, global_id 16056 2024-05-15T13:20:31.498 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.498+0000 7ff4084ca640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T13:20:31.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.498+0000 7ff405264640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:20:31.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.498+0000 7ff4084ca640 10 monclient: get_monmap_and_config success 2024-05-15T13:20:31.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.498+0000 7ff405264640 1 -- 172.21.15.117:0/2783724844 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f80343f0 con 0x7ff4000af440 2024-05-15T13:20:31.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.498+0000 7ff405264640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:31.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.498+0000 7ff4084ca640 4 set_mon_vals no callback set 2024-05-15T13:20:31.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.498+0000 7ff405264640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T13:20:31.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.498+0000 7ff405264640 10 monclient: dump: 2024-05-15T13:20:31.499 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:31.499 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:31.499 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:31.499 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:31.499 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:31.499 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:31.499 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:31.499 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:31.500 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:31.500 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:31.503 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.499+0000 7ff4084ca640 10 monclient: shutdown 2024-05-15T13:20:31.503 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.499+0000 7ff4084ca640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:20:31.503 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.499+0000 7ff4084ca640 1 -- 172.21.15.117:0/2783724844 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff4000af440 msgr2=0x7ff400146590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:31.503 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.499+0000 7ff4084ca640 1 --2- 172.21.15.117:0/2783724844 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff4000af440 0x7ff400146590 secure :-1 s=READY pgs=3301 cs=0 l=1 rev1=1 crypto rx=0x7ff400072c00 tx=0x7ff3f802bee0 comp rx=0 tx=0).stop 2024-05-15T13:20:31.503 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.499+0000 7ff4084ca640 1 -- 172.21.15.117:0/2783724844 shutdown_connections 2024-05-15T13:20:31.503 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.499+0000 7ff4084ca640 1 --2- 172.21.15.117:0/2783724844 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff4000af440 0x7ff400146590 unknown :-1 s=CLOSED pgs=3301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:31.504 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.499+0000 7ff4084ca640 1 --2- 172.21.15.117:0/2783724844 >> v2:172.21.15.178:3300/0 conn(0x7ff40006c900 0x7ff4000aeab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:31.504 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.499+0000 7ff4084ca640 1 -- 172.21.15.117:0/2783724844 >> 172.21.15.117:0/2783724844 conn(0x7ff4000b2ac0 msgr2=0x7ff4000b2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:20:31.504 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.499+0000 7ff4084ca640 1 -- 172.21.15.117:0/2783724844 shutdown_connections 2024-05-15T13:20:31.504 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.499+0000 7ff4084ca640 1 -- 172.21.15.117:0/2783724844 wait complete. 2024-05-15T13:20:31.504 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.500+0000 7ff4084ca640 10 monclient: build_initial_monmap 2024-05-15T13:20:31.504 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.500+0000 7ff4084ca640 10 monclient: monmap: 2024-05-15T13:20:31.504 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T13:20:31.504 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T13:20:31.504 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-05-15T13:20:31.504 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-05-15T13:20:31.504 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T13:20:31.504 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:31.504 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T13:20:31.504 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T13:20:31.504 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T13:20:31.504 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:31.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.500+0000 7ff4084ca640 1 librados: starting msgr at 2024-05-15T13:20:31.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.500+0000 7ff4084ca640 1 librados: starting objecter 2024-05-15T13:20:31.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.500+0000 7ff4084ca640 1 Processor -- start 2024-05-15T13:20:31.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.500+0000 7ff4084ca640 1 -- start start 2024-05-15T13:20:31.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.500+0000 7ff4084ca640 1 librados: setting wanted keys 2024-05-15T13:20:31.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.500+0000 7ff4084ca640 1 librados: calling monclient init 2024-05-15T13:20:31.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.500+0000 7ff4084ca640 10 monclient: init 2024-05-15T13:20:31.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.501+0000 7ff4084ca640 10 monclient: _reopen_session rank -1 2024-05-15T13:20:31.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.501+0000 7ff4084ca640 10 monclient: _add_conns ranks=[0,1,2] 2024-05-15T13:20:31.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.501+0000 7ff4084ca640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff40006c900 0x7ff400121e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:31.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.501+0000 7ff4084ca640 10 monclient(hunting): picked mon.noname-a con 0x7ff40006c900 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T13:20:31.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.501+0000 7ff4084ca640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff4000aeff0 0x7ff4001210b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:31.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.501+0000 7ff4084ca640 10 monclient(hunting): picked mon.noname-b con 0x7ff4000aeff0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T13:20:31.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.501+0000 7ff4084ca640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff4000af440 0x7ff4001d7f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:31.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.501+0000 7ff4084ca640 10 monclient(hunting): picked mon.noname-c con 0x7ff4000af440 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T13:20:31.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.501+0000 7ff4084ca640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:31.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.501+0000 7ff4084ca640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff400146ad0 con 0x7ff40006c900 2024-05-15T13:20:31.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.501+0000 7ff4084ca640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:31.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.501+0000 7ff4084ca640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff40006d730 con 0x7ff4000aeff0 2024-05-15T13:20:31.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.501+0000 7ff4084ca640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:31.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.501+0000 7ff4084ca640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff400146c50 con 0x7ff4000af440 2024-05-15T13:20:31.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.501+0000 7ff4084ca640 10 monclient(hunting): _renew_subs 2024-05-15T13:20:31.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.501+0000 7ff4084ca640 10 monclient(hunting): authenticate will time out at 3594.743144s 2024-05-15T13:20:31.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.501+0000 7ff406266640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff40006c900 0x7ff400121e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:31.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.501+0000 7ff406a67640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff4000af440 0x7ff4001d7f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:31.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.502+0000 7ff405a65640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff4000aeff0 0x7ff4001210b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:31.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.502+0000 7ff406266640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff40006c900 0x7ff400121e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:44260/0 (socket says 172.21.15.117:44260) 2024-05-15T13:20:31.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.502+0000 7ff406266640 1 -- 172.21.15.117:0/847741526 learned_addr learned my addr 172.21.15.117:0/847741526 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T13:20:31.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.502+0000 7ff406266640 10 monclient(hunting): get_auth_request con 0x7ff40006c900 auth_method 0 2024-05-15T13:20:31.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.502+0000 7ff406266640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:31.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.502+0000 7ff406266640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:31.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.502+0000 7ff406266640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:31.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.502+0000 7ff406a67640 10 monclient(hunting): get_auth_request con 0x7ff4000af440 auth_method 0 2024-05-15T13:20:31.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.502+0000 7ff406a67640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:31.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.502+0000 7ff406a67640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:31.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.502+0000 7ff406a67640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:31.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.502+0000 7ff405a65640 10 monclient(hunting): get_auth_request con 0x7ff4000aeff0 auth_method 0 2024-05-15T13:20:31.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.502+0000 7ff405a65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:31.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.502+0000 7ff405a65640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:31.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.502+0000 7ff405a65640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:31.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.502+0000 7ff406266640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:31.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.502+0000 7ff406266640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:31.508 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.502+0000 7ff406266640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:31.508 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.502+0000 7ff405a65640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:31.508 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.502+0000 7ff405a65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:31.508 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.502+0000 7ff405a65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:31.508 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.502+0000 7ff406a67640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:31.508 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.502+0000 7ff406a67640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:31.508 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.502+0000 7ff406a67640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:31.510 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.505+0000 7ff406266640 10 monclient(hunting): handle_auth_done global_id 16059 payload 824 2024-05-15T13:20:31.510 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.505+0000 7ff406266640 1 -- 172.21.15.117:0/847741526 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff4000af440 msgr2=0x7ff4001d7f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:31.510 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.505+0000 7ff406266640 1 --2- 172.21.15.117:0/847741526 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff4000af440 0x7ff4001d7f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:31.510 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.505+0000 7ff406266640 1 -- 172.21.15.117:0/847741526 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff4000aeff0 msgr2=0x7ff4001210b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:31.510 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.505+0000 7ff406266640 1 --2- 172.21.15.117:0/847741526 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff4000aeff0 0x7ff4001210b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:31.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.505+0000 7ff406266640 10 monclient: _finish_hunting 0 2024-05-15T13:20:31.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.505+0000 7ff406266640 1 monclient: found mon.noname-a 2024-05-15T13:20:31.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.505+0000 7ff406266640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:20:31.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.505+0000 7ff406266640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.117:3300/0 2024-05-15T13:20:31.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.505+0000 7ff406266640 1 -- 172.21.15.117:0/847741526 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff400123580 con 0x7ff40006c900 2024-05-15T13:20:31.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.507+0000 7ff406266640 1 --2- 172.21.15.117:0/847741526 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff40006c900 0x7ff400121e50 secure :-1 s=READY pgs=3302 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc00e9b0 tx=0x7ff3fc00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:31.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.507+0000 7ff3f6ffd640 10 client.?.objecter ms_handle_connect 0x7ff40006c900 2024-05-15T13:20:31.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.507+0000 7ff3f6ffd640 10 client.?.objecter resend_mon_ops 2024-05-15T13:20:31.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.509+0000 7ff3f6ffd640 1 -- 172.21.15.117:0/847741526 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3fc017070 con 0x7ff40006c900 2024-05-15T13:20:31.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.509+0000 7ff3f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:31.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.509+0000 7ff3f6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T13:20:31.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.509+0000 7ff3f6ffd640 10 monclient: dump: 2024-05-15T13:20:31.511 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:31.511 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:31.511 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:31.511 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:31.512 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:31.512 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:31.512 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:31.512 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:31.512 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:31.512 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:31.512 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.509+0000 7ff3f6ffd640 10 monclient: _finish_auth 0 2024-05-15T13:20:31.512 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.509+0000 7ff3f6ffd640 10 monclient: _check_auth_tickets 2024-05-15T13:20:31.512 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.509+0000 7ff3f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:20:31.512 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.509+0000 7ff4084ca640 5 monclient: authenticate success, global_id 16059 2024-05-15T13:20:31.512 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.509+0000 7ff4084ca640 10 monclient: _renew_subs 2024-05-15T13:20:31.512 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.509+0000 7ff4084ca640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T13:20:31.512 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.509+0000 7ff4084ca640 1 -- 172.21.15.117:0/847741526 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4001d79e0 con 0x7ff40006c900 2024-05-15T13:20:31.512 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.509+0000 7ff4084ca640 10 client.16059.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:20:31.512 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.509+0000 7ff4084ca640 10 monclient: _renew_subs 2024-05-15T13:20:31.512 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.509+0000 7ff4084ca640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T13:20:31.512 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.509+0000 7ff4084ca640 1 -- 172.21.15.117:0/847741526 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4001d8440 con 0x7ff40006c900 2024-05-15T13:20:31.513 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.509+0000 7ff4084ca640 1 librados: init done 2024-05-15T13:20:31.513 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.510+0000 7ff4084ca640 10 client.16059.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:20:31.513 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.510+0000 7ff4084ca640 15 client.16059.objecter _session_command_op_assign -1 1 2024-05-15T13:20:31.513 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.510+0000 7ff4084ca640 20 client.16059.objecter _get_session osd=-1 returning homeless 2024-05-15T13:20:31.513 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.510+0000 7ff4084ca640 10 client.16059.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:20:31.513 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.510+0000 7ff4084ca640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T13:20:31.513 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.510+0000 7ff4084ca640 1 -- 172.21.15.117:0/847741526 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff40004f0e0 con 0x7ff40006c900 2024-05-15T13:20:31.513 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.511+0000 7ff3f6ffd640 1 -- 172.21.15.117:0/847741526 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff40004f0e0 con 0x7ff40006c900 2024-05-15T13:20:31.513 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.511+0000 7ff3f6ffd640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:20:31.513 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.511+0000 7ff3f6ffd640 1 -- 172.21.15.117:0/847741526 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3fc021060 con 0x7ff40006c900 2024-05-15T13:20:31.513 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.511+0000 7ff3f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:31.513 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.511+0000 7ff3f6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T13:20:31.513 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.511+0000 7ff3f6ffd640 10 monclient: dump: 2024-05-15T13:20:31.513 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:31.513 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:31.513 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:31.514 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:31.514 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:31.514 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:31.514 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:31.514 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:31.514 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:31.514 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:31.514 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.511+0000 7ff404a63640 4 set_mon_vals no callback set 2024-05-15T13:20:31.529 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.531+0000 7ff3f6ffd640 1 -- 172.21.15.117:0/847741526 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7ff3fc01a070 con 0x7ff40006c900 2024-05-15T13:20:31.530 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.531+0000 7ff3f6ffd640 1 --2- 172.21.15.117:0/847741526 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7ff3e406ce60 0x7ff3e406f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:31.530 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.531+0000 7ff405a65640 1 --2- 172.21.15.117:0/847741526 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7ff3e406ce60 0x7ff3e406f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:31.531 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.532+0000 7ff405a65640 10 monclient: get_auth_request con 0x7ff3e406ce60 auth_method 0 2024-05-15T13:20:31.531 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.532+0000 7ff405a65640 1 --2- 172.21.15.117:0/847741526 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7ff3e406ce60 0x7ff3e406f320 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7ff400146130 tx=0x7ff3f0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:31.531 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.532+0000 7ff3f6ffd640 10 client.16059.objecter ms_handle_connect 0x7ff3e406ce60 2024-05-15T13:20:31.534 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.535+0000 7ff3f6ffd640 1 -- 172.21.15.117:0/847741526 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(1225..1225 src has 912..1225) v4 ==== 6365+0+0 (secure 0 0 0) 0x7ff3fc057040 con 0x7ff40006c900 2024-05-15T13:20:31.534 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.535+0000 7ff3f6ffd640 10 client.16059.objecter ms_dispatch 0x7ff400000b90 osd_map(1225..1225 src has 912..1225) v4 2024-05-15T13:20:31.534 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.535+0000 7ff3f6ffd640 3 client.16059.objecter handle_osd_map got epochs [1225,1225] > 0 2024-05-15T13:20:31.534 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.535+0000 7ff3f6ffd640 3 client.16059.objecter handle_osd_map decoding full epoch 1225 2024-05-15T13:20:31.534 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.535+0000 7ff3f6ffd640 10 client.16059.objecter checking command 1 2024-05-15T13:20:31.534 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.535+0000 7ff3f6ffd640 1 --2- 172.21.15.117:0/847741526 >> [v2:172.21.15.178:6824/248472851,v1:172.21.15.178:6825/248472851] conn(0x7ff3e4074b70 0x7ff3e4077010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:31.534 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.535+0000 7ff3f6ffd640 20 client.16059.objecter _get_session s=0x7ff3e40744a0 osd=4 3 2024-05-15T13:20:31.534 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.535+0000 7ff3f6ffd640 20 client.16059.objecter put_session s=0x7ff3e40744a0 osd=4 3 2024-05-15T13:20:31.534 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.535+0000 7ff3f6ffd640 15 client.16059.objecter _session_command_op_remove -1 1 2024-05-15T13:20:31.534 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.535+0000 7ff3f6ffd640 20 client.16059.objecter _get_session s=0x7ff3e40744a0 osd=4 3 2024-05-15T13:20:31.534 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.535+0000 7ff3f6ffd640 20 client.16059.objecter get_session s=0x7ff3e40744a0 osd=4 3 2024-05-15T13:20:31.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.535+0000 7ff3f6ffd640 15 client.16059.objecter _session_command_op_assign 4 1 2024-05-15T13:20:31.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.535+0000 7ff3f6ffd640 20 client.16059.objecter put_session s=0x7ff3e40744a0 osd=4 4 2024-05-15T13:20:31.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.535+0000 7ff3f6ffd640 10 client.16059.objecter _send_command 1 2024-05-15T13:20:31.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.535+0000 7ff3f6ffd640 1 -- 172.21.15.117:0/847741526 --> [v2:172.21.15.178:6824/248472851,v1:172.21.15.178:6825/248472851] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff3e40776e0 con 0x7ff3e4074b70 2024-05-15T13:20:31.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.535+0000 7ff3f6ffd640 20 client.16059.objecter dump_active .. 0 homeless 2024-05-15T13:20:31.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.535+0000 7ff3f6ffd640 1 -- 172.21.15.117:0/847741526 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_get_version_reply(handle=1 version=1225) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff3fc055240 con 0x7ff40006c900 2024-05-15T13:20:31.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.535+0000 7ff3f6ffd640 10 monclient: handle_get_version_reply finishing 1 version 1225 2024-05-15T13:20:31.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.536+0000 7ff406a67640 1 --2- 172.21.15.117:0/847741526 >> [v2:172.21.15.178:6824/248472851,v1:172.21.15.178:6825/248472851] conn(0x7ff3e4074b70 0x7ff3e4077010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:31.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.536+0000 7ff406a67640 10 monclient: get_auth_request con 0x7ff3e4074b70 auth_method 0 2024-05-15T13:20:31.537 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.539+0000 7ff406a67640 1 --2- 172.21.15.117:0/847741526 >> [v2:172.21.15.178:6824/248472851,v1:172.21.15.178:6825/248472851] conn(0x7ff3e4074b70 0x7ff3e4077010 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:31.538 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.539+0000 7ff3f6ffd640 10 client.16059.objecter ms_handle_connect 0x7ff3e4074b70 2024-05-15T13:20:31.561 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.562+0000 7ff3f6ffd640 1 -- 172.21.15.117:0/847741526 <== osd.4 v2:172.21.15.178:6824/248472851 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff3e40776e0 con 0x7ff3e4074b70 2024-05-15T13:20:31.561 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.562+0000 7ff3f6ffd640 10 client.16059.objecter ms_dispatch 0x7ff400000b90 command_reply(tid 1: 0 ) v1 2024-05-15T13:20:31.561 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.562+0000 7ff3f6ffd640 10 client.16059.objecter _finish_command 1 = system:0 2024-05-15T13:20:31.561 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.562+0000 7ff3f6ffd640 20 client.16059.objecter put_session s=0x7ff3e40744a0 osd=4 4 2024-05-15T13:20:31.561 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.562+0000 7ff3f6ffd640 15 client.16059.objecter _session_command_op_remove 4 1 2024-05-15T13:20:31.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.575+0000 7ff4084ca640 10 client.16059.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-05-15T13:20:31.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.575+0000 7ff4084ca640 15 client.16059.objecter _session_command_op_assign -1 2 2024-05-15T13:20:31.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.575+0000 7ff4084ca640 20 client.16059.objecter _get_session s=0x7ff3e40744a0 osd=4 3 2024-05-15T13:20:31.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.575+0000 7ff4084ca640 20 client.16059.objecter put_session s=0x7ff3e40744a0 osd=4 3 2024-05-15T13:20:31.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.575+0000 7ff4084ca640 20 client.16059.objecter _get_session s=0x7ff3e40744a0 osd=4 3 2024-05-15T13:20:31.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.575+0000 7ff4084ca640 15 client.16059.objecter _session_command_op_remove -1 2 2024-05-15T13:20:31.575 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.575+0000 7ff4084ca640 20 client.16059.objecter get_session s=0x7ff3e40744a0 osd=4 3 2024-05-15T13:20:31.575 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.575+0000 7ff4084ca640 15 client.16059.objecter _session_command_op_assign 4 2 2024-05-15T13:20:31.575 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.575+0000 7ff4084ca640 20 client.16059.objecter put_session s=0x7ff3e40744a0 osd=4 4 2024-05-15T13:20:31.575 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.575+0000 7ff4084ca640 10 client.16059.objecter _send_command 2 2024-05-15T13:20:31.575 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.575+0000 7ff4084ca640 1 -- 172.21.15.117:0/847741526 --> [v2:172.21.15.178:6824/248472851,v1:172.21.15.178:6825/248472851] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7ff4000b5630 con 0x7ff3e4074b70 2024-05-15T13:20:31.587 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.588+0000 7ff3f6ffd640 1 -- 172.21.15.117:0/847741526 <== osd.4 v2:172.21.15.178:6824/248472851 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7ff4000b5630 con 0x7ff3e4074b70 2024-05-15T13:20:31.587 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.588+0000 7ff3f6ffd640 10 client.16059.objecter ms_dispatch 0x7ff400000b90 command_reply(tid 2: 0 ) v1 2024-05-15T13:20:31.587 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.588+0000 7ff3f6ffd640 10 client.16059.objecter _finish_command 2 = system:0 2024-05-15T13:20:31.588 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.588+0000 7ff3f6ffd640 20 client.16059.objecter put_session s=0x7ff3e40744a0 osd=4 4 2024-05-15T13:20:31.588 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.588+0000 7ff3f6ffd640 15 client.16059.objecter _session_command_op_remove 4 2 2024-05-15T13:20:31.588 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-05-15T13:20:31.588 INFO:teuthology.orchestra.run.smithi117.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-15T13:20:31.588 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-05-15T13:20:31.590 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.589+0000 7ff4084ca640 10 librados: watch_flush enter 2024-05-15T13:20:31.590 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.589+0000 7ff4084ca640 10 librados: watch_flush exit 2024-05-15T13:20:31.590 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.589+0000 7ff4084ca640 10 client.16059.objecter close_session for osd.4 2024-05-15T13:20:31.590 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.589+0000 7ff4084ca640 1 -- 172.21.15.117:0/847741526 >> [v2:172.21.15.178:6824/248472851,v1:172.21.15.178:6825/248472851] conn(0x7ff3e4074b70 msgr2=0x7ff3e4077010 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:31.590 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.589+0000 7ff4084ca640 1 --2- 172.21.15.117:0/847741526 >> [v2:172.21.15.178:6824/248472851,v1:172.21.15.178:6825/248472851] conn(0x7ff3e4074b70 0x7ff3e4077010 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:31.590 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.589+0000 7ff4084ca640 20 client.16059.objecter put_session s=0x7ff3e40744a0 osd=4 1 2024-05-15T13:20:31.590 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.589+0000 7ff4084ca640 20 client.16059.objecter shutdown clearing up homeless session... 2024-05-15T13:20:31.590 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.589+0000 7ff4084ca640 10 client.16059.objecter successfully canceled tick 2024-05-15T13:20:31.590 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.589+0000 7ff4084ca640 1 -- 172.21.15.117:0/847741526 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7ff3e406ce60 msgr2=0x7ff3e406f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:31.590 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.589+0000 7ff4084ca640 1 --2- 172.21.15.117:0/847741526 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7ff3e406ce60 0x7ff3e406f320 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7ff400146130 tx=0x7ff3f0009040 comp rx=0 tx=0).stop 2024-05-15T13:20:31.590 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.589+0000 7ff4084ca640 10 monclient: shutdown 2024-05-15T13:20:31.590 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.589+0000 7ff4084ca640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:20:31.590 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.589+0000 7ff4084ca640 1 -- 172.21.15.117:0/847741526 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff40006c900 msgr2=0x7ff400121e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:31.591 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.589+0000 7ff4084ca640 1 --2- 172.21.15.117:0/847741526 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff40006c900 0x7ff400121e50 secure :-1 s=READY pgs=3302 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc00e9b0 tx=0x7ff3fc00ee70 comp rx=0 tx=0).stop 2024-05-15T13:20:31.591 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.589+0000 7ff406a67640 1 -- 172.21.15.117:0/847741526 reap_dead start 2024-05-15T13:20:31.591 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.589+0000 7ff4084ca640 1 -- 172.21.15.117:0/847741526 shutdown_connections 2024-05-15T13:20:31.591 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.589+0000 7ff4084ca640 1 -- 172.21.15.117:0/847741526 >> 172.21.15.117:0/847741526 conn(0x7ff4000b2ac0 msgr2=0x7ff4000b64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:20:31.591 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.589+0000 7ff4084ca640 1 -- 172.21.15.117:0/847741526 shutdown_connections 2024-05-15T13:20:31.591 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.589+0000 7ff4084ca640 1 -- 172.21.15.117:0/847741526 wait complete. 2024-05-15T13:20:31.591 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.589+0000 7ff4084ca640 1 librados: shutdown 2024-05-15T13:20:31.599 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-05-15T13:20:31.702 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.703+0000 7f6fd8490640 10 monclient: get_monmap_and_config 2024-05-15T13:20:31.702 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.703+0000 7f6fd8490640 10 monclient: build_initial_monmap 2024-05-15T13:20:31.702 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.703+0000 7f6fd8490640 10 monclient: monmap: 2024-05-15T13:20:31.703 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T13:20:31.703 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:31.703 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T13:20:31.704604+0000 2024-05-15T13:20:31.703 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T13:20:31.704604+0000 2024-05-15T13:20:31.703 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T13:20:31.703 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:31.703 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T13:20:31.703 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T13:20:31.703 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T13:20:31.703 INFO:teuthology.orchestra.run.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T13:20:31.703 INFO:teuthology.orchestra.run.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T13:20:31.703 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:31.703 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.704+0000 7f6fd8490640 1 Processor -- start 2024-05-15T13:20:31.704 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.704+0000 7f6fd8490640 1 -- start start 2024-05-15T13:20:31.704 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.704+0000 7f6fd8490640 10 monclient: init 2024-05-15T13:20:31.705 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.704+0000 7f6fd8490640 10 monclient: _reopen_session rank -1 2024-05-15T13:20:31.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.704+0000 7f6fd8490640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-05-15T13:20:31.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.704+0000 7f6fd8490640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6fd00b4d00 addr v1:172.21.15.178:6789/0 2024-05-15T13:20:31.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.704+0000 7f6fd8490640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f6fd00b50e0 0x7f6fd00b1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:31.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.704+0000 7f6fd8490640 10 monclient(hunting): picked mon.noname-a con 0x7f6fd00b50e0 addr v2:172.21.15.117:3300/0 2024-05-15T13:20:31.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.705+0000 7f6fd8490640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f6fd00a94b0 0x7f6fd00b0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:31.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.705+0000 7f6fd8490640 10 monclient(hunting): picked mon.noname-c con 0x7f6fd00a94b0 addr v2:172.21.15.178:3300/0 2024-05-15T13:20:31.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.705+0000 7f6fd8490640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fd00ab1b0 con 0x7f6fd00b4d00 2024-05-15T13:20:31.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.705+0000 7f6fd8490640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:31.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.705+0000 7f6fd8490640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fd00b23c0 con 0x7f6fd00b50e0 2024-05-15T13:20:31.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.705+0000 7f6fd8490640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:31.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.705+0000 7f6fd8490640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fd00b1520 con 0x7f6fd00a94b0 2024-05-15T13:20:31.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.705+0000 7f6fd8490640 10 monclient(hunting): _renew_subs 2024-05-15T13:20:31.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.705+0000 7f6fd8490640 10 monclient(hunting): authenticate will time out at 3594.946851s 2024-05-15T13:20:31.707 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.705+0000 7f6fd5a2b640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f6fd00b50e0 0x7f6fd00b1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:31.707 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.705+0000 7f6fd5a2b640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f6fd00b50e0 0x7f6fd00b1e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:44276/0 (socket says 172.21.15.117:44276) 2024-05-15T13:20:31.707 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.705+0000 7f6fd5a2b640 1 -- 172.21.15.117:0/2298048435 learned_addr learned my addr 172.21.15.117:0/2298048435 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T13:20:31.707 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.705+0000 7f6fd5a2b640 10 monclient(hunting): get_auth_request con 0x7f6fd00b50e0 auth_method 0 2024-05-15T13:20:31.707 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.705+0000 7f6fd5a2b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:31.707 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.705+0000 7f6fd5a2b640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:31.707 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.705+0000 7f6fd5a2b640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:31.707 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.705+0000 7f6fd6a2d640 1 --2- 172.21.15.117:0/2298048435 >> v2:172.21.15.178:3300/0 conn(0x7f6fd00a94b0 0x7f6fd00b0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:31.707 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.706+0000 7f6fd5a2b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:31.707 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.706+0000 7f6fd5a2b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:31.707 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.706+0000 7f6fd5a2b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:31.707 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.706+0000 7f6fd6a2d640 10 monclient(hunting): get_auth_request con 0x7f6fd00a94b0 auth_method 0 2024-05-15T13:20:31.707 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.706+0000 7f6fd6a2d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:31.707 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.706+0000 7f6fd6a2d640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:31.707 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.706+0000 7f6fd6a2d640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:31.708 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.707+0000 7f6fd6a2d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:31.708 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.707+0000 7f6fd6a2d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:31.708 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.707+0000 7f6fd6a2d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:31.708 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.708+0000 7f6fd5a2b640 10 monclient(hunting): handle_auth_done global_id 16062 payload 274 2024-05-15T13:20:31.708 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.708+0000 7f6fd5a2b640 1 -- 172.21.15.117:0/2298048435 >> v2:172.21.15.178:3300/0 conn(0x7f6fd00a94b0 msgr2=0x7f6fd00b0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T13:20:31.709 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.708+0000 7f6fd5a2b640 1 --2- 172.21.15.117:0/2298048435 >> v2:172.21.15.178:3300/0 conn(0x7f6fd00a94b0 0x7f6fd00b0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:31.709 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.708+0000 7f6fd5a2b640 1 -- 172.21.15.117:0/2298048435 >> v1:172.21.15.178:6789/0 conn(0x7f6fd00b4d00 legacy=0x7f6fd0145c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:31.709 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.708+0000 7f6fd5a2b640 10 monclient: _finish_hunting 0 2024-05-15T13:20:31.709 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.708+0000 7f6fd5a2b640 1 monclient: found mon.noname-a 2024-05-15T13:20:31.709 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.708+0000 7f6fd5a2b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:20:31.709 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.708+0000 7f6fd5a2b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.117:3300/0 2024-05-15T13:20:31.709 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.708+0000 7f6fd5a2b640 1 -- 172.21.15.117:0/2298048435 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd00b16d0 con 0x7f6fd00b50e0 2024-05-15T13:20:31.709 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.710+0000 7f6fd5a2b640 1 --2- 172.21.15.117:0/2298048435 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6fd00b50e0 0x7f6fd00b1e80 secure :-1 s=READY pgs=3303 cs=0 l=1 rev1=1 crypto rx=0x7f6fd00aa590 tx=0x7f6fc002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14cd9e10bb52bae8 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:31.712 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.713+0000 7f6fd522a640 1 -- 172.21.15.117:0/2298048435 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc0035020 con 0x7f6fd00b50e0 2024-05-15T13:20:31.712 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.713+0000 7f6fd522a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:31.712 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.713+0000 7f6fd522a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T13:20:31.712 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.713+0000 7f6fd522a640 10 monclient: dump: 2024-05-15T13:20:31.712 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:31.712 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:31.712 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:31.712 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:31.712 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:31.712 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:31.713 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:31.713 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:31.713 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:31.713 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:31.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.713+0000 7f6fd522a640 10 monclient: _finish_auth 0 2024-05-15T13:20:31.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.713+0000 7f6fd522a640 10 monclient: _check_auth_tickets 2024-05-15T13:20:31.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.713+0000 7f6fd522a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:20:31.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.713+0000 7f6fd8490640 5 monclient: authenticate success, global_id 16062 2024-05-15T13:20:31.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.713+0000 7f6fd522a640 1 -- 172.21.15.117:0/2298048435 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fc0031870 con 0x7f6fd00b50e0 2024-05-15T13:20:31.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.713+0000 7f6fd8490640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T13:20:31.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.713+0000 7f6fd522a640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:20:31.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.713+0000 7f6fd8490640 10 monclient: get_monmap_and_config success 2024-05-15T13:20:31.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.713+0000 7f6fd8490640 4 set_mon_vals no callback set 2024-05-15T13:20:31.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.713+0000 7f6fd522a640 1 -- 172.21.15.117:0/2298048435 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc0031c40 con 0x7f6fd00b50e0 2024-05-15T13:20:31.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.713+0000 7f6fd522a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:31.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.713+0000 7f6fd522a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T13:20:31.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.713+0000 7f6fd522a640 10 monclient: dump: 2024-05-15T13:20:31.714 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:31.714 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:31.714 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:31.714 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:31.714 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:31.714 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:31.714 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:31.714 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:31.714 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:31.714 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:31.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.714+0000 7f6fd8490640 10 monclient: shutdown 2024-05-15T13:20:31.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.714+0000 7f6fd8490640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:20:31.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.714+0000 7f6fd8490640 1 -- 172.21.15.117:0/2298048435 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6fd00b50e0 msgr2=0x7f6fd00b1e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:31.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.714+0000 7f6fd8490640 1 --2- 172.21.15.117:0/2298048435 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6fd00b50e0 0x7f6fd00b1e80 secure :-1 s=READY pgs=3303 cs=0 l=1 rev1=1 crypto rx=0x7f6fd00aa590 tx=0x7f6fc002b910 comp rx=0 tx=0).stop 2024-05-15T13:20:31.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.714+0000 7f6fd8490640 1 -- 172.21.15.117:0/2298048435 shutdown_connections 2024-05-15T13:20:31.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.714+0000 7f6fd8490640 1 --2- 172.21.15.117:0/2298048435 >> v2:172.21.15.178:3300/0 conn(0x7f6fd00a94b0 0x7f6fd00b0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:31.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.714+0000 7f6fd8490640 1 --2- 172.21.15.117:0/2298048435 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6fd00b50e0 0x7f6fd00b1e80 unknown :-1 s=CLOSED pgs=3303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:31.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.714+0000 7f6fd8490640 1 -- 172.21.15.117:0/2298048435 >> 172.21.15.117:0/2298048435 conn(0x7f6fd0066cc0 msgr2=0x7f6fd00670d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:20:31.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.714+0000 7f6fd8490640 1 -- 172.21.15.117:0/2298048435 shutdown_connections 2024-05-15T13:20:31.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.714+0000 7f6fd8490640 1 -- 172.21.15.117:0/2298048435 wait complete. 2024-05-15T13:20:31.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.715+0000 7f6fd8490640 10 monclient: build_initial_monmap 2024-05-15T13:20:31.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.715+0000 7f6fd8490640 10 monclient: monmap: 2024-05-15T13:20:31.719 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T13:20:31.719 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T13:20:31.719 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-05-15T13:20:31.719 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-05-15T13:20:31.719 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T13:20:31.719 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:31.719 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T13:20:31.719 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T13:20:31.719 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T13:20:31.719 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:31.719 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.715+0000 7f6fd8490640 1 librados: starting msgr at 2024-05-15T13:20:31.719 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.715+0000 7f6fd8490640 1 librados: starting objecter 2024-05-15T13:20:31.719 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.716+0000 7f6fd8490640 1 Processor -- start 2024-05-15T13:20:31.719 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.716+0000 7f6fd8490640 1 -- start start 2024-05-15T13:20:31.719 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.716+0000 7f6fd8490640 1 librados: setting wanted keys 2024-05-15T13:20:31.720 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.716+0000 7f6fd8490640 1 librados: calling monclient init 2024-05-15T13:20:31.720 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.716+0000 7f6fd8490640 10 monclient: init 2024-05-15T13:20:31.720 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.716+0000 7f6fd8490640 10 monclient: _reopen_session rank -1 2024-05-15T13:20:31.720 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.716+0000 7f6fd8490640 10 monclient: _add_conns ranks=[1,2,0] 2024-05-15T13:20:31.720 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.716+0000 7f6fd8490640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6fd00a94b0 0x7f6fd0142a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:31.720 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.717+0000 7f6fd8490640 10 monclient(hunting): picked mon.noname-b con 0x7f6fd00a94b0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T13:20:31.720 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.717+0000 7f6fd8490640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6fd00b4d00 0x7f6fd01d7d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:31.720 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.717+0000 7f6fd8490640 10 monclient(hunting): picked mon.noname-c con 0x7f6fd00b4d00 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T13:20:31.720 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.717+0000 7f6fd8490640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6fd01dafe0 0x7f6fd01db490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:31.720 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.717+0000 7f6fd8490640 10 monclient(hunting): picked mon.noname-a con 0x7f6fd01dafe0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T13:20:31.720 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.717+0000 7f6fd8490640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:31.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.717+0000 7f6fd8490640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd014cfb0 con 0x7f6fd01dafe0 2024-05-15T13:20:31.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.717+0000 7f6fd8490640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:31.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.717+0000 7f6fd8490640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd00a9b70 con 0x7f6fd00a94b0 2024-05-15T13:20:31.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.717+0000 7f6fd8490640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:31.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.717+0000 7f6fd8490640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd014d130 con 0x7f6fd00b4d00 2024-05-15T13:20:31.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.717+0000 7f6fd8490640 10 monclient(hunting): _renew_subs 2024-05-15T13:20:31.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.717+0000 7f6fd8490640 10 monclient(hunting): authenticate will time out at 3594.958876s 2024-05-15T13:20:31.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.717+0000 7f6fd6a2d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6fd01dafe0 0x7f6fd01db490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:31.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.717+0000 7f6fd5a2b640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6fd00b4d00 0x7f6fd01d7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:31.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.717+0000 7f6fd622c640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6fd00a94b0 0x7f6fd0142a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:31.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.717+0000 7f6fd6a2d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6fd01dafe0 0x7f6fd01db490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:44278/0 (socket says 172.21.15.117:44278) 2024-05-15T13:20:31.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.717+0000 7f6fd6a2d640 1 -- 172.21.15.117:0/3466757891 learned_addr learned my addr 172.21.15.117:0/3466757891 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T13:20:31.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.717+0000 7f6fd6a2d640 10 monclient(hunting): get_auth_request con 0x7f6fd01dafe0 auth_method 0 2024-05-15T13:20:31.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.717+0000 7f6fd6a2d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:31.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.717+0000 7f6fd6a2d640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:31.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.717+0000 7f6fd6a2d640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:31.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.717+0000 7f6fd622c640 10 monclient(hunting): get_auth_request con 0x7f6fd00a94b0 auth_method 0 2024-05-15T13:20:31.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.717+0000 7f6fd622c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:31.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.717+0000 7f6fd622c640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:31.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.717+0000 7f6fd622c640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:31.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.718+0000 7f6fd5a2b640 10 monclient(hunting): get_auth_request con 0x7f6fd00b4d00 auth_method 0 2024-05-15T13:20:31.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.718+0000 7f6fd5a2b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:31.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.718+0000 7f6fd5a2b640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:31.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.718+0000 7f6fd5a2b640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:31.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.718+0000 7f6fd6a2d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:31.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.718+0000 7f6fd6a2d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:31.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.718+0000 7f6fd6a2d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:31.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.718+0000 7f6fd622c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:31.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.718+0000 7f6fd622c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:31.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.718+0000 7f6fd622c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:31.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.718+0000 7f6fd5a2b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:31.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.718+0000 7f6fd5a2b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:31.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.718+0000 7f6fd5a2b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:31.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.720+0000 7f6fd622c640 10 monclient(hunting): handle_auth_done global_id 22186 payload 824 2024-05-15T13:20:31.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.720+0000 7f6fd622c640 1 -- 172.21.15.117:0/3466757891 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6fd00b4d00 msgr2=0x7f6fd01d7d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:31.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.720+0000 7f6fd622c640 1 --2- 172.21.15.117:0/3466757891 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6fd00b4d00 0x7f6fd01d7d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:31.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.720+0000 7f6fd622c640 1 -- 172.21.15.117:0/3466757891 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6fd01dafe0 msgr2=0x7f6fd01db490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:31.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.720+0000 7f6fd622c640 1 --2- 172.21.15.117:0/3466757891 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6fd01dafe0 0x7f6fd01db490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:31.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.720+0000 7f6fd622c640 10 monclient: _finish_hunting 0 2024-05-15T13:20:31.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.720+0000 7f6fd622c640 1 monclient: found mon.noname-b 2024-05-15T13:20:31.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.720+0000 7f6fd622c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:20:31.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.720+0000 7f6fd622c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2024-05-15T13:20:31.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.720+0000 7f6fd622c640 1 -- 172.21.15.117:0/3466757891 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd00abe60 con 0x7f6fd00a94b0 2024-05-15T13:20:31.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.722+0000 7f6fd622c640 1 --2- 172.21.15.117:0/3466757891 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6fd00a94b0 0x7f6fd0142a90 secure :-1 s=READY pgs=5453 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc00c3c0 tx=0x7f6fcc00c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:31.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.722+0000 7f6fc6ffd640 10 client.?.objecter ms_handle_connect 0x7f6fd00a94b0 2024-05-15T13:20:31.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.723+0000 7f6fc6ffd640 10 client.?.objecter resend_mon_ops 2024-05-15T13:20:31.726 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.726+0000 7f6fc6ffd640 1 -- 172.21.15.117:0/3466757891 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fcc017070 con 0x7f6fd00a94b0 2024-05-15T13:20:31.726 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.726+0000 7f6fc6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:31.726 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.726+0000 7f6fc6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T13:20:31.726 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.726+0000 7f6fc6ffd640 10 monclient: dump: 2024-05-15T13:20:31.726 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:31.726 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:31.726 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:31.726 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:31.726 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:31.727 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:31.727 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:31.727 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:31.727 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:31.727 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:31.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.726+0000 7f6fc6ffd640 10 monclient: _finish_auth 0 2024-05-15T13:20:31.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.726+0000 7f6fc6ffd640 10 monclient: _check_auth_tickets 2024-05-15T13:20:31.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.726+0000 7f6fc6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:20:31.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.726+0000 7f6fc6ffd640 1 -- 172.21.15.117:0/3466757891 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fcc0052d0 con 0x7f6fd00a94b0 2024-05-15T13:20:31.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.726+0000 7f6fd8490640 5 monclient: authenticate success, global_id 22186 2024-05-15T13:20:31.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.726+0000 7f6fd8490640 10 monclient: _renew_subs 2024-05-15T13:20:31.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.726+0000 7f6fd8490640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T13:20:31.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.726+0000 7f6fd8490640 1 -- 172.21.15.117:0/3466757891 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fd01d86e0 con 0x7f6fd00a94b0 2024-05-15T13:20:31.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.726+0000 7f6fc6ffd640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:20:31.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.726+0000 7f6fc6ffd640 1 -- 172.21.15.117:0/3466757891 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fcc003830 con 0x7f6fd00a94b0 2024-05-15T13:20:31.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.726+0000 7f6fc6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:31.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.726+0000 7f6fc6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T13:20:31.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.726+0000 7f6fc6ffd640 10 monclient: dump: 2024-05-15T13:20:31.728 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:31.728 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:31.728 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:31.728 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:31.728 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:31.728 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:31.728 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:31.728 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:31.728 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:31.728 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:31.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.726+0000 7f6fd4a29640 4 set_mon_vals no callback set 2024-05-15T13:20:31.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.726+0000 7f6fd8490640 10 client.22186.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:20:31.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.726+0000 7f6fd8490640 10 monclient: _renew_subs 2024-05-15T13:20:31.729 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.726+0000 7f6fd8490640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T13:20:31.729 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.726+0000 7f6fd8490640 1 -- 172.21.15.117:0/3466757891 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fd01db9d0 con 0x7f6fd00a94b0 2024-05-15T13:20:31.729 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.726+0000 7f6fd8490640 1 librados: init done 2024-05-15T13:20:31.729 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.727+0000 7f6fd8490640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T13:20:31.729 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.727+0000 7f6fd8490640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:20:31.729 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.727+0000 7f6fd8490640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T13:20:31.729 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.727+0000 7f6fd8490640 1 -- 172.21.15.117:0/3466757891 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fd01dbcc0 con 0x7f6fd00a94b0 2024-05-15T13:20:31.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.745+0000 7f6fc6ffd640 1 -- 172.21.15.117:0/3466757891 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f6fcc01a070 con 0x7f6fd00a94b0 2024-05-15T13:20:31.744 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.745+0000 7f6fc6ffd640 1 --2- 172.21.15.117:0/3466757891 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f6fb406ceb0 0x7f6fb406f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:31.744 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.745+0000 7f6fd5a2b640 1 --2- 172.21.15.117:0/3466757891 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f6fb406ceb0 0x7f6fb406f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:31.744 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.745+0000 7f6fd5a2b640 10 monclient: get_auth_request con 0x7f6fb406ceb0 auth_method 0 2024-05-15T13:20:31.744 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.746+0000 7f6fd5a2b640 1 --2- 172.21.15.117:0/3466757891 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f6fb406ceb0 0x7f6fb406f370 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f6fc0036000 tx=0x7f6fc003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:31.744 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.746+0000 7f6fc6ffd640 10 client.22186.objecter ms_handle_connect 0x7f6fb406ceb0 2024-05-15T13:20:31.746 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.747+0000 7f6fc6ffd640 1 -- 172.21.15.117:0/3466757891 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(1225..1225 src has 912..1225) v4 ==== 6365+0+0 (secure 0 0 0) 0x7f6fcc0561d0 con 0x7f6fd00a94b0 2024-05-15T13:20:31.746 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.748+0000 7f6fc6ffd640 10 client.22186.objecter ms_dispatch 0x7f6fd0000b90 osd_map(1225..1225 src has 912..1225) v4 2024-05-15T13:20:31.746 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.748+0000 7f6fc6ffd640 3 client.22186.objecter handle_osd_map got epochs [1225,1225] > 0 2024-05-15T13:20:31.746 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.748+0000 7f6fc6ffd640 3 client.22186.objecter handle_osd_map decoding full epoch 1225 2024-05-15T13:20:31.747 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.748+0000 7f6fc6ffd640 20 client.22186.objecter dump_active .. 0 homeless 2024-05-15T13:20:31.985 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.985+0000 7f6fc6ffd640 1 -- 172.21.15.117:0/3466757891 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f6fcc021020 con 0x7f6fd00a94b0 2024-05-15T13:20:31.985 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.986+0000 7f6fc6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:20:31.985 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:31.986+0000 7f6fc6ffd640 10 monclient: _finish_command 1 = system:0 2024-05-15T13:20:32.167 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.168+0000 7f6fd8490640 1 -- 172.21.15.117:0/3466757891 --> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f6fd004f0e0 con 0x7f6fb406ceb0 2024-05-15T13:20:32.167 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.168+0000 7f6fc6ffd640 1 -- 172.21.15.117:0/3466757891 <== mgr.4107 v2:172.21.15.117:6800/1785766523 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f6fd004f0e0 con 0x7f6fb406ceb0 2024-05-15T13:20:32.167 INFO:teuthology.orchestra.run.smithi117.stderr:instructed osd(s) 4 to deep-scrub 2024-05-15T13:20:32.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.170+0000 7f6fd8490640 10 librados: watch_flush enter 2024-05-15T13:20:32.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.170+0000 7f6fd8490640 10 librados: watch_flush exit 2024-05-15T13:20:32.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.170+0000 7f6fd8490640 20 client.22186.objecter shutdown clearing up homeless session... 2024-05-15T13:20:32.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.170+0000 7f6fd8490640 10 client.22186.objecter successfully canceled tick 2024-05-15T13:20:32.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.170+0000 7f6fd8490640 1 -- 172.21.15.117:0/3466757891 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f6fb406ceb0 msgr2=0x7f6fb406f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:32.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.170+0000 7f6fd8490640 1 --2- 172.21.15.117:0/3466757891 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f6fb406ceb0 0x7f6fb406f370 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f6fc0036000 tx=0x7f6fc003c040 comp rx=0 tx=0).stop 2024-05-15T13:20:32.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.170+0000 7f6fd8490640 10 monclient: shutdown 2024-05-15T13:20:32.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.170+0000 7f6fd8490640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:20:32.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.170+0000 7f6fd8490640 1 -- 172.21.15.117:0/3466757891 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6fd00a94b0 msgr2=0x7f6fd0142a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:32.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.170+0000 7f6fd8490640 1 --2- 172.21.15.117:0/3466757891 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6fd00a94b0 0x7f6fd0142a90 secure :-1 s=READY pgs=5453 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc00c3c0 tx=0x7f6fcc00c880 comp rx=0 tx=0).stop 2024-05-15T13:20:32.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.170+0000 7f6fd8490640 1 -- 172.21.15.117:0/3466757891 shutdown_connections 2024-05-15T13:20:32.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.170+0000 7f6fd8490640 1 --2- 172.21.15.117:0/3466757891 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6fd01dafe0 0x7f6fd01db490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:32.171 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.170+0000 7f6fd8490640 1 --2- 172.21.15.117:0/3466757891 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6fd00b4d00 0x7f6fd01d7d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:32.171 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.170+0000 7f6fd8490640 1 --2- 172.21.15.117:0/3466757891 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f6fb406ceb0 0x7f6fb406f370 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:32.171 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.170+0000 7f6fd8490640 1 --2- 172.21.15.117:0/3466757891 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6fd00a94b0 0x7f6fd0142a90 unknown :-1 s=CLOSED pgs=5453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:32.171 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.170+0000 7f6fd8490640 1 -- 172.21.15.117:0/3466757891 >> 172.21.15.117:0/3466757891 conn(0x7f6fd0066cc0 msgr2=0x7f6fd00b86c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:20:32.171 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.170+0000 7f6fd8490640 1 -- 172.21.15.117:0/3466757891 shutdown_connections 2024-05-15T13:20:32.171 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.171+0000 7f6fd8490640 1 -- 172.21.15.117:0/3466757891 wait complete. 2024-05-15T13:20:32.171 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.171+0000 7f6fd8490640 1 librados: shutdown 2024-05-15T13:20:32.180 INFO:tasks.ceph:Scrubbing osd.5 2024-05-15T13:20:32.181 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-05-15T13:20:32.301 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.303+0000 7f6e21bde640 10 monclient: get_monmap_and_config 2024-05-15T13:20:32.301 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.303+0000 7f6e21bde640 10 monclient: build_initial_monmap 2024-05-15T13:20:32.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.303+0000 7f6e21bde640 10 monclient: monmap: 2024-05-15T13:20:32.302 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T13:20:32.302 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:32.302 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T13:20:32.304071+0000 2024-05-15T13:20:32.302 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T13:20:32.304071+0000 2024-05-15T13:20:32.302 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T13:20:32.302 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:32.302 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T13:20:32.302 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T13:20:32.302 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T13:20:32.302 INFO:teuthology.orchestra.run.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T13:20:32.302 INFO:teuthology.orchestra.run.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T13:20:32.302 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:32.304 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.303+0000 7f6e21bde640 1 Processor -- start 2024-05-15T13:20:32.304 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.303+0000 7f6e21bde640 1 -- start start 2024-05-15T13:20:32.304 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.303+0000 7f6e21bde640 10 monclient: init 2024-05-15T13:20:32.304 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.304+0000 7f6e21bde640 10 monclient: _reopen_session rank -1 2024-05-15T13:20:32.304 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.304+0000 7f6e21bde640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-05-15T13:20:32.304 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.304+0000 7f6e21bde640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6e1c0b5180 addr v1:172.21.15.178:6789/0 2024-05-15T13:20:32.304 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.304+0000 7f6e21bde640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6e1c13ff40 addr v1:172.21.15.117:6789/0 2024-05-15T13:20:32.304 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.304+0000 7f6e21bde640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f6e1c142d90 0x7f6e1c0a9980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:32.304 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.304+0000 7f6e21bde640 10 monclient(hunting): picked mon.noname-a con 0x7f6e1c142d90 addr v2:172.21.15.117:3300/0 2024-05-15T13:20:32.304 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.304+0000 7f6e21bde640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e1c0aa530 con 0x7f6e1c13ff40 2024-05-15T13:20:32.304 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.304+0000 7f6e21bde640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e1c0ab6b0 con 0x7f6e1c0b5180 2024-05-15T13:20:32.305 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.304+0000 7f6e21bde640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:32.305 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.304+0000 7f6e21bde640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e1c0b5560 con 0x7f6e1c142d90 2024-05-15T13:20:32.305 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.304+0000 7f6e21bde640 10 monclient(hunting): _renew_subs 2024-05-15T13:20:32.305 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.304+0000 7f6e21bde640 10 monclient(hunting): authenticate will time out at 3595.546732s 2024-05-15T13:20:32.305 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.305+0000 7f6e1b7fe640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f6e1c0b5180 0x7f6e1c13f830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.117:36840/0 (socket says 172.21.15.117:36840) 2024-05-15T13:20:32.305 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.305+0000 7f6e1b7fe640 1 -- 172.21.15.117:0/1578411677 learned_addr learned my addr 172.21.15.117:0/1578411677 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-05-15T13:20:32.305 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.307+0000 7f6e1a7fc640 1 -- 172.21.15.117:0/1578411677 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1282614921 0 0) 0x7f6e1c0ab6b0 con 0x7f6e1c0b5180 2024-05-15T13:20:32.305 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.307+0000 7f6e1a7fc640 10 monclient(hunting): _negotiate 2024-05-15T13:20:32.306 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.307+0000 7f6e1a7fc640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:32.306 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.307+0000 7f6e1a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:32.306 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.307+0000 7f6e1a7fc640 10 monclient(hunting): my global_id is 22189 2024-05-15T13:20:32.306 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.307+0000 7f6e1a7fc640 1 -- 172.21.15.117:0/1578411677 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e04003520 con 0x7f6e1c0b5180 2024-05-15T13:20:32.306 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.308+0000 7f6e1bfff640 1 --2- 172.21.15.117:0/1578411677 >> v2:172.21.15.117:3300/0 conn(0x7f6e1c142d90 0x7f6e1c0a9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:32.307 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.308+0000 7f6e1bfff640 10 monclient(hunting): get_auth_request con 0x7f6e1c142d90 auth_method 0 2024-05-15T13:20:32.307 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.308+0000 7f6e1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:32.307 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.308+0000 7f6e1bfff640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:32.307 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.308+0000 7f6e1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:32.308 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.310+0000 7f6e1bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:32.309 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.310+0000 7f6e1bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:32.309 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.310+0000 7f6e1bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:32.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.311+0000 7f6e1a7fc640 1 -- 172.21.15.117:0/1578411677 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1163588728 0 0) 0x7f6e04003520 con 0x7f6e1c0b5180 2024-05-15T13:20:32.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.311+0000 7f6e1a7fc640 1 -- 172.21.15.117:0/1578411677 >> v2:172.21.15.117:3300/0 conn(0x7f6e1c142d90 msgr2=0x7f6e1c0a9980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T13:20:32.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.311+0000 7f6e1a7fc640 1 --2- 172.21.15.117:0/1578411677 >> v2:172.21.15.117:3300/0 conn(0x7f6e1c142d90 0x7f6e1c0a9980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:32.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.311+0000 7f6e1a7fc640 1 -- 172.21.15.117:0/1578411677 >> v1:172.21.15.117:6789/0 conn(0x7f6e1c13ff40 legacy=0x7f6e1c142680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:32.312 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.311+0000 7f6e1a7fc640 10 monclient: _finish_hunting 0 2024-05-15T13:20:32.312 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.311+0000 7f6e1a7fc640 1 monclient: found mon.noname-c-legacy 2024-05-15T13:20:32.312 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.311+0000 7f6e1a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:20:32.312 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.311+0000 7f6e1a7fc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.178:6789/0 2024-05-15T13:20:32.312 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.311+0000 7f6e1a7fc640 1 -- 172.21.15.117:0/1578411677 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e1c0aa010 con 0x7f6e1c0b5180 2024-05-15T13:20:32.312 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.311+0000 7f6e1a7fc640 10 monclient: _finish_auth 0 2024-05-15T13:20:32.312 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.311+0000 7f6e1a7fc640 10 monclient: _check_auth_tickets 2024-05-15T13:20:32.312 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.311+0000 7f6e1a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:20:32.312 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.311+0000 7f6e1a7fc640 1 -- 172.21.15.117:0/1578411677 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3166367242 0 0) 0x7f6e0c001670 con 0x7f6e1c0b5180 2024-05-15T13:20:32.312 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.311+0000 7f6e1a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:32.312 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.311+0000 7f6e1a7fc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-05-15T13:20:32.312 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.311+0000 7f6e1a7fc640 10 monclient: dump: 2024-05-15T13:20:32.312 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:32.312 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:32.312 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:32.313 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:32.313 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:32.313 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:32.313 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:32.313 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:32.313 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:32.313 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:32.313 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.311+0000 7f6e1a7fc640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] but i'm connected to v1:172.21.15.178:6789/0, reconnecting 2024-05-15T13:20:32.313 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.311+0000 7f6e1a7fc640 10 monclient: _reopen_session rank -1 2024-05-15T13:20:32.313 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.311+0000 7f6e1a7fc640 1 -- 172.21.15.117:0/1578411677 >> v1:172.21.15.178:6789/0 conn(0x7f6e1c0b5180 legacy=0x7f6e1c13f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:32.313 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.311+0000 7f6e1a7fc640 10 monclient: _add_conns ranks=[2,0,1] 2024-05-15T13:20:32.313 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.311+0000 7f6e1a7fc640 1 --2- 172.21.15.117:0/1578411677 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6e04003a00 0x7f6e04003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:32.313 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.311+0000 7f6e1a7fc640 10 monclient(hunting): picked mon.c con 0x7f6e04003a00 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T13:20:32.313 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.311+0000 7f6e1a7fc640 1 --2- 172.21.15.117:0/1578411677 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e04004800 0x7f6e04004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:32.313 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.311+0000 7f6e1a7fc640 10 monclient(hunting): picked mon.a con 0x7f6e04004800 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T13:20:32.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.311+0000 7f6e1a7fc640 1 --2- 172.21.15.117:0/1578411677 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6e04005210 0x7f6e0400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:32.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.311+0000 7f6e1a7fc640 10 monclient(hunting): picked mon.b con 0x7f6e04005210 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T13:20:32.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.311+0000 7f6e1a7fc640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:32.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.311+0000 7f6e1a7fc640 1 -- 172.21.15.117:0/1578411677 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e1c0b5560 con 0x7f6e04004800 2024-05-15T13:20:32.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.311+0000 7f6e1a7fc640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:32.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.311+0000 7f6e1a7fc640 1 -- 172.21.15.117:0/1578411677 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0400c160 con 0x7f6e04005210 2024-05-15T13:20:32.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.311+0000 7f6e1a7fc640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:32.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.311+0000 7f6e1a7fc640 1 -- 172.21.15.117:0/1578411677 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0400c360 con 0x7f6e04003a00 2024-05-15T13:20:32.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.311+0000 7f6e1a7fc640 10 monclient(hunting): _renew_subs 2024-05-15T13:20:32.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.311+0000 7f6e1a7fc640 10 monclient(hunting): _finish_auth 0 2024-05-15T13:20:32.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.312+0000 7f6e1b7fe640 1 --2- 172.21.15.117:0/1578411677 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6e04003a00 0x7f6e04003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:32.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.313+0000 7f6e1b7fe640 10 monclient(hunting): get_auth_request con 0x7f6e04003a00 auth_method 0 2024-05-15T13:20:32.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.313+0000 7f6e1b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:32.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.313+0000 7f6e1b7fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:32.315 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.313+0000 7f6e1b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-05-15T13:20:32.316 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.313+0000 7f6e1b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:32.316 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.313+0000 7f6e1b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:32.316 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.313+0000 7f6e1b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-05-15T13:20:32.316 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.313+0000 7f6e1bfff640 1 --2- 172.21.15.117:0/1578411677 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6e04005210 0x7f6e0400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:32.316 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.315+0000 7f6e1bfff640 10 monclient(hunting): get_auth_request con 0x7f6e04005210 auth_method 0 2024-05-15T13:20:32.316 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.315+0000 7f6e1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:32.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.315+0000 7f6e1bfff640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:32.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.315+0000 7f6e1bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-05-15T13:20:32.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.315+0000 7f6e1bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:32.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.315+0000 7f6e1bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:32.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.316+0000 7f6e1bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-05-15T13:20:32.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.316+0000 7f6e1b7fe640 10 monclient(hunting): handle_auth_done global_id 22189 payload 293 2024-05-15T13:20:32.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.316+0000 7f6e1b7fe640 1 -- 172.21.15.117:0/1578411677 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6e04005210 msgr2=0x7f6e0400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T13:20:32.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.316+0000 7f6e1b7fe640 1 --2- 172.21.15.117:0/1578411677 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6e04005210 0x7f6e0400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:32.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.316+0000 7f6e1b7fe640 1 -- 172.21.15.117:0/1578411677 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e04004800 msgr2=0x7f6e04004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T13:20:32.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.316+0000 7f6e1b7fe640 1 --2- 172.21.15.117:0/1578411677 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e04004800 0x7f6e04004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:32.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.316+0000 7f6e1b7fe640 10 monclient: _finish_hunting 0 2024-05-15T13:20:32.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.316+0000 7f6e1b7fe640 1 monclient: found mon.c 2024-05-15T13:20:32.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.316+0000 7f6e1b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:20:32.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.316+0000 7f6e1b7fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.117:3301/0 2024-05-15T13:20:32.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.316+0000 7f6e1bfff640 1 -- 172.21.15.117:0/1578411677 reap_dead start 2024-05-15T13:20:32.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.316+0000 7f6e1b7fe640 1 -- 172.21.15.117:0/1578411677 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e04003520 con 0x7f6e04003a00 2024-05-15T13:20:32.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.316+0000 7f6e1b7fe640 10 monclient: _finish_auth 0 2024-05-15T13:20:32.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.316+0000 7f6e1b7fe640 10 monclient: _check_auth_tickets 2024-05-15T13:20:32.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.316+0000 7f6e1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:20:32.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.316+0000 7f6e21bde640 5 monclient: authenticate success, global_id 22189 2024-05-15T13:20:32.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.316+0000 7f6e21bde640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T13:20:32.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.318+0000 7f6e1b7fe640 1 --2- 172.21.15.117:0/1578411677 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6e04003a00 0x7f6e04003e20 secure :-1 s=READY pgs=3386 cs=0 l=1 rev1=1 crypto rx=0x7f6e0c00c0e0 tx=0x7f6e04005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb8d1e996d773909 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:32.321 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.322+0000 7f6e1a7fc640 1 -- 172.21.15.117:0/1578411677 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e0c007c40 con 0x7f6e04003a00 2024-05-15T13:20:32.321 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.322+0000 7f6e1a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:32.321 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.322+0000 7f6e1a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T13:20:32.321 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.322+0000 7f6e1a7fc640 10 monclient: dump: 2024-05-15T13:20:32.321 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:32.321 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:32.321 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:32.321 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:32.321 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:32.321 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:32.322 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:32.322 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:32.322 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:32.322 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:32.322 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.322+0000 7f6e1a7fc640 1 -- 172.21.15.117:0/1578411677 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e0c030070 con 0x7f6e04003a00 2024-05-15T13:20:32.322 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.322+0000 7f6e1a7fc640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:20:32.322 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.322+0000 7f6e1a7fc640 1 -- 172.21.15.117:0/1578411677 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e0c005db0 con 0x7f6e04003a00 2024-05-15T13:20:32.322 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.322+0000 7f6e21bde640 10 monclient: get_monmap_and_config success 2024-05-15T13:20:32.322 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.322+0000 7f6e21bde640 4 set_mon_vals no callback set 2024-05-15T13:20:32.322 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.322+0000 7f6e1a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:32.322 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.322+0000 7f6e1a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T13:20:32.322 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.322+0000 7f6e1a7fc640 10 monclient: dump: 2024-05-15T13:20:32.322 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:32.322 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:32.323 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:32.323 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:32.323 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:32.323 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:32.323 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:32.323 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:32.323 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:32.323 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:32.326 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.323+0000 7f6e21bde640 10 monclient: shutdown 2024-05-15T13:20:32.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.323+0000 7f6e21bde640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:20:32.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.323+0000 7f6e21bde640 1 -- 172.21.15.117:0/1578411677 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6e04003a00 msgr2=0x7f6e04003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:32.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.323+0000 7f6e21bde640 1 --2- 172.21.15.117:0/1578411677 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6e04003a00 0x7f6e04003e20 secure :-1 s=READY pgs=3386 cs=0 l=1 rev1=1 crypto rx=0x7f6e0c00c0e0 tx=0x7f6e04005830 comp rx=0 tx=0).stop 2024-05-15T13:20:32.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.323+0000 7f6e21bde640 1 -- 172.21.15.117:0/1578411677 shutdown_connections 2024-05-15T13:20:32.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.323+0000 7f6e21bde640 1 --2- 172.21.15.117:0/1578411677 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6e04003a00 0x7f6e04003e20 unknown :-1 s=CLOSED pgs=3386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:32.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.323+0000 7f6e21bde640 1 -- 172.21.15.117:0/1578411677 >> 172.21.15.117:0/1578411677 conn(0x7f6e1c066ed0 msgr2=0x7f6e1c0672e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:20:32.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.323+0000 7f6e21bde640 1 -- 172.21.15.117:0/1578411677 shutdown_connections 2024-05-15T13:20:32.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.323+0000 7f6e21bde640 1 -- 172.21.15.117:0/1578411677 wait complete. 2024-05-15T13:20:32.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.324+0000 7f6e21bde640 10 monclient: build_initial_monmap 2024-05-15T13:20:32.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.324+0000 7f6e21bde640 10 monclient: monmap: 2024-05-15T13:20:32.327 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T13:20:32.327 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T13:20:32.327 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-05-15T13:20:32.328 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-05-15T13:20:32.328 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T13:20:32.328 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:32.328 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T13:20:32.328 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T13:20:32.328 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T13:20:32.328 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:32.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.324+0000 7f6e21bde640 1 librados: starting msgr at 2024-05-15T13:20:32.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.324+0000 7f6e21bde640 1 librados: starting objecter 2024-05-15T13:20:32.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.324+0000 7f6e21bde640 1 Processor -- start 2024-05-15T13:20:32.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.324+0000 7f6e21bde640 1 -- start start 2024-05-15T13:20:32.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.324+0000 7f6e21bde640 1 librados: setting wanted keys 2024-05-15T13:20:32.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.324+0000 7f6e21bde640 1 librados: calling monclient init 2024-05-15T13:20:32.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.324+0000 7f6e21bde640 10 monclient: init 2024-05-15T13:20:32.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.325+0000 7f6e21bde640 10 monclient: _reopen_session rank -1 2024-05-15T13:20:32.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.325+0000 7f6e21bde640 10 monclient: _add_conns ranks=[1,2,0] 2024-05-15T13:20:32.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.325+0000 7f6e21bde640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6e04003a00 0x7f6e1c0b8830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:32.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.325+0000 7f6e21bde640 10 monclient(hunting): picked mon.noname-b con 0x7f6e04003a00 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T13:20:32.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.325+0000 7f6e21bde640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6e1c1ca870 0x7f6e1c1cbad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:32.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.325+0000 7f6e21bde640 10 monclient(hunting): picked mon.noname-c con 0x7f6e1c1ca870 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T13:20:32.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.325+0000 7f6e21bde640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e1c1db0d0 0x7f6e1c1c87f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:32.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.325+0000 7f6e21bde640 10 monclient(hunting): picked mon.noname-a con 0x7f6e1c1db0d0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T13:20:32.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.325+0000 7f6e21bde640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:32.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.325+0000 7f6e21bde640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e1c0a9b00 con 0x7f6e1c1db0d0 2024-05-15T13:20:32.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.325+0000 7f6e21bde640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:32.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.325+0000 7f6e21bde640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e1c0a9980 con 0x7f6e04003a00 2024-05-15T13:20:32.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.325+0000 7f6e21bde640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:32.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.325+0000 7f6e21bde640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e1c0a9c80 con 0x7f6e1c1ca870 2024-05-15T13:20:32.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.325+0000 7f6e21bde640 10 monclient(hunting): _renew_subs 2024-05-15T13:20:32.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.325+0000 7f6e21bde640 10 monclient(hunting): authenticate will time out at 3595.567095s 2024-05-15T13:20:32.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.325+0000 7f6e1b7fe640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6e04003a00 0x7f6e1c0b8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:32.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.325+0000 7f6e1bfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e1c1db0d0 0x7f6e1c1c87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:32.330 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.325+0000 7f6e1affd640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6e1c1ca870 0x7f6e1c1cbad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:32.330 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.326+0000 7f6e1b7fe640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6e04003a00 0x7f6e1c0b8830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:37408/0 (socket says 172.21.15.117:37408) 2024-05-15T13:20:32.330 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.326+0000 7f6e1b7fe640 1 -- 172.21.15.117:0/3101673290 learned_addr learned my addr 172.21.15.117:0/3101673290 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T13:20:32.330 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.326+0000 7f6e1b7fe640 10 monclient(hunting): get_auth_request con 0x7f6e04003a00 auth_method 0 2024-05-15T13:20:32.330 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.326+0000 7f6e1b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:32.330 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.326+0000 7f6e1b7fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:32.330 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.326+0000 7f6e1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:32.330 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.326+0000 7f6e1bfff640 10 monclient(hunting): get_auth_request con 0x7f6e1c1db0d0 auth_method 0 2024-05-15T13:20:32.330 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.326+0000 7f6e1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:32.330 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.326+0000 7f6e1bfff640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:32.330 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.326+0000 7f6e1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:32.330 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.326+0000 7f6e1affd640 10 monclient(hunting): get_auth_request con 0x7f6e1c1ca870 auth_method 0 2024-05-15T13:20:32.330 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.326+0000 7f6e1affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:32.330 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.326+0000 7f6e1affd640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:32.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.326+0000 7f6e1affd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:32.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.326+0000 7f6e1b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:32.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.326+0000 7f6e1b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:32.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.326+0000 7f6e1b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:32.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.326+0000 7f6e1bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:32.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.326+0000 7f6e1bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:32.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.326+0000 7f6e1bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:32.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.326+0000 7f6e1affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:32.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.326+0000 7f6e1affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:32.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.326+0000 7f6e1affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:32.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.329+0000 7f6e1b7fe640 10 monclient(hunting): handle_auth_done global_id 22192 payload 824 2024-05-15T13:20:32.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.329+0000 7f6e1b7fe640 1 -- 172.21.15.117:0/3101673290 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6e1c1ca870 msgr2=0x7f6e1c1cbad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:32.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.329+0000 7f6e1b7fe640 1 --2- 172.21.15.117:0/3101673290 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6e1c1ca870 0x7f6e1c1cbad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:32.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.329+0000 7f6e1b7fe640 1 -- 172.21.15.117:0/3101673290 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e1c1db0d0 msgr2=0x7f6e1c1c87f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:32.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.329+0000 7f6e1b7fe640 1 --2- 172.21.15.117:0/3101673290 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e1c1db0d0 0x7f6e1c1c87f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:32.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.329+0000 7f6e1b7fe640 10 monclient: _finish_hunting 0 2024-05-15T13:20:32.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.329+0000 7f6e1b7fe640 1 monclient: found mon.noname-b 2024-05-15T13:20:32.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.329+0000 7f6e1b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:20:32.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.329+0000 7f6e1b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2024-05-15T13:20:32.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.329+0000 7f6e1b7fe640 1 -- 172.21.15.117:0/3101673290 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e1c1ccbf0 con 0x7f6e04003a00 2024-05-15T13:20:32.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.331+0000 7f6e1b7fe640 1 --2- 172.21.15.117:0/3101673290 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6e04003a00 0x7f6e1c0b8830 secure :-1 s=READY pgs=5455 cs=0 l=1 rev1=1 crypto rx=0x7f6e0c00c200 tx=0x7f6e1c0ab880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:32.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.331+0000 7f6e03fff640 10 client.?.objecter ms_handle_connect 0x7f6e04003a00 2024-05-15T13:20:32.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.331+0000 7f6e03fff640 10 client.?.objecter resend_mon_ops 2024-05-15T13:20:32.336 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.336+0000 7f6e03fff640 1 -- 172.21.15.117:0/3101673290 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e0c045070 con 0x7f6e04003a00 2024-05-15T13:20:32.336 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.336+0000 7f6e03fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:32.336 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.336+0000 7f6e03fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T13:20:32.336 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.336+0000 7f6e03fff640 10 monclient: dump: 2024-05-15T13:20:32.336 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:32.336 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:32.336 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:32.336 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:32.337 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:32.337 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:32.337 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:32.337 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:32.337 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:32.337 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:32.337 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.336+0000 7f6e03fff640 10 monclient: _finish_auth 0 2024-05-15T13:20:32.337 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.336+0000 7f6e03fff640 10 monclient: _check_auth_tickets 2024-05-15T13:20:32.337 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.336+0000 7f6e03fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:20:32.337 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.336+0000 7f6e03fff640 1 -- 172.21.15.117:0/3101673290 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e0c000ed0 con 0x7f6e04003a00 2024-05-15T13:20:32.337 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.336+0000 7f6e03fff640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:20:32.337 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.336+0000 7f6e03fff640 1 -- 172.21.15.117:0/3101673290 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e0c007680 con 0x7f6e04003a00 2024-05-15T13:20:32.337 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.336+0000 7f6e03fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:32.337 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.336+0000 7f6e03fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T13:20:32.337 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.336+0000 7f6e03fff640 10 monclient: dump: 2024-05-15T13:20:32.338 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:32.338 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:32.338 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:32.338 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:32.338 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:32.338 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:32.338 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:32.338 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:32.338 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:32.338 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:32.338 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.337+0000 7f6e21bde640 5 monclient: authenticate success, global_id 22192 2024-05-15T13:20:32.338 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.337+0000 7f6e21bde640 10 monclient: _renew_subs 2024-05-15T13:20:32.338 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.337+0000 7f6e21bde640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T13:20:32.338 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.337+0000 7f6e21bde640 1 -- 172.21.15.117:0/3101673290 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e1c1cced0 con 0x7f6e04003a00 2024-05-15T13:20:32.339 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.336+0000 7f6e19ffb640 4 set_mon_vals no callback set 2024-05-15T13:20:32.339 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.337+0000 7f6e21bde640 10 client.22192.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:20:32.339 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.337+0000 7f6e21bde640 10 monclient: _renew_subs 2024-05-15T13:20:32.339 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.337+0000 7f6e21bde640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T13:20:32.339 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.337+0000 7f6e21bde640 1 -- 172.21.15.117:0/3101673290 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e1c1cb260 con 0x7f6e04003a00 2024-05-15T13:20:32.339 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.337+0000 7f6e21bde640 1 librados: init done 2024-05-15T13:20:32.339 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.337+0000 7f6e21bde640 10 client.22192.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:20:32.339 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.337+0000 7f6e21bde640 15 client.22192.objecter _session_command_op_assign -1 1 2024-05-15T13:20:32.339 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.337+0000 7f6e21bde640 20 client.22192.objecter _get_session osd=-1 returning homeless 2024-05-15T13:20:32.339 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.337+0000 7f6e21bde640 10 client.22192.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:20:32.339 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.337+0000 7f6e21bde640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T13:20:32.339 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.337+0000 7f6e21bde640 1 -- 172.21.15.117:0/3101673290 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6e1c04f0e0 con 0x7f6e04003a00 2024-05-15T13:20:32.356 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.357+0000 7f6e03fff640 1 -- 172.21.15.117:0/3101673290 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f6e0c035470 con 0x7f6e04003a00 2024-05-15T13:20:32.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.358+0000 7f6e03fff640 1 --2- 172.21.15.117:0/3101673290 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f6e0806ce60 0x7f6e0806f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:32.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.358+0000 7f6e1affd640 1 --2- 172.21.15.117:0/3101673290 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f6e0806ce60 0x7f6e0806f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:32.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.358+0000 7f6e1affd640 10 monclient: get_auth_request con 0x7f6e0806ce60 auth_method 0 2024-05-15T13:20:32.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.358+0000 7f6e1affd640 1 --2- 172.21.15.117:0/3101673290 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f6e0806ce60 0x7f6e0806f320 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f6e10007fd0 tx=0x7f6e10009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:32.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.358+0000 7f6e03fff640 10 client.22192.objecter ms_handle_connect 0x7f6e0806ce60 2024-05-15T13:20:32.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.360+0000 7f6e03fff640 1 -- 172.21.15.117:0/3101673290 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(1225..1225 src has 912..1225) v4 ==== 6365+0+0 (secure 0 0 0) 0x7f6e0c078510 con 0x7f6e04003a00 2024-05-15T13:20:32.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.360+0000 7f6e03fff640 10 client.22192.objecter ms_dispatch 0x7f6e1c000b90 osd_map(1225..1225 src has 912..1225) v4 2024-05-15T13:20:32.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.360+0000 7f6e03fff640 3 client.22192.objecter handle_osd_map got epochs [1225,1225] > 0 2024-05-15T13:20:32.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.360+0000 7f6e03fff640 3 client.22192.objecter handle_osd_map decoding full epoch 1225 2024-05-15T13:20:32.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.360+0000 7f6e03fff640 10 client.22192.objecter checking command 1 2024-05-15T13:20:32.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.360+0000 7f6e03fff640 1 --2- 172.21.15.117:0/3101673290 >> [v2:172.21.15.178:6816/3713764498,v1:172.21.15.178:6817/3713764498] conn(0x7f6e08074b70 0x7f6e08077010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:32.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.360+0000 7f6e03fff640 20 client.22192.objecter _get_session s=0x7f6e080744a0 osd=5 3 2024-05-15T13:20:32.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.360+0000 7f6e03fff640 20 client.22192.objecter put_session s=0x7f6e080744a0 osd=5 3 2024-05-15T13:20:32.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.360+0000 7f6e03fff640 15 client.22192.objecter _session_command_op_remove -1 1 2024-05-15T13:20:32.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.360+0000 7f6e03fff640 20 client.22192.objecter _get_session s=0x7f6e080744a0 osd=5 3 2024-05-15T13:20:32.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.360+0000 7f6e03fff640 20 client.22192.objecter get_session s=0x7f6e080744a0 osd=5 3 2024-05-15T13:20:32.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.360+0000 7f6e03fff640 15 client.22192.objecter _session_command_op_assign 5 1 2024-05-15T13:20:32.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.360+0000 7f6e03fff640 20 client.22192.objecter put_session s=0x7f6e080744a0 osd=5 4 2024-05-15T13:20:32.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.360+0000 7f6e03fff640 10 client.22192.objecter _send_command 1 2024-05-15T13:20:32.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.360+0000 7f6e03fff640 1 -- 172.21.15.117:0/3101673290 --> [v2:172.21.15.178:6816/3713764498,v1:172.21.15.178:6817/3713764498] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6e0c00bf30 con 0x7f6e08074b70 2024-05-15T13:20:32.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.360+0000 7f6e03fff640 20 client.22192.objecter dump_active .. 0 homeless 2024-05-15T13:20:32.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.360+0000 7f6e03fff640 1 -- 172.21.15.117:0/3101673290 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_get_version_reply(handle=1 version=1225) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6e0c076710 con 0x7f6e04003a00 2024-05-15T13:20:32.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.360+0000 7f6e03fff640 10 monclient: handle_get_version_reply finishing 1 version 1225 2024-05-15T13:20:32.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.361+0000 7f6e1bfff640 1 --2- 172.21.15.117:0/3101673290 >> [v2:172.21.15.178:6816/3713764498,v1:172.21.15.178:6817/3713764498] conn(0x7f6e08074b70 0x7f6e08077010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:32.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.361+0000 7f6e1bfff640 10 monclient: get_auth_request con 0x7f6e08074b70 auth_method 0 2024-05-15T13:20:32.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.364+0000 7f6e03fff640 10 client.22192.objecter ms_handle_connect 0x7f6e08074b70 2024-05-15T13:20:32.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.364+0000 7f6e1bfff640 1 --2- 172.21.15.117:0/3101673290 >> [v2:172.21.15.178:6816/3713764498,v1:172.21.15.178:6817/3713764498] conn(0x7f6e08074b70 0x7f6e08077010 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:32.386 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.387+0000 7f6e03fff640 1 -- 172.21.15.117:0/3101673290 <== osd.5 v2:172.21.15.178:6816/3713764498 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6e0c00bf30 con 0x7f6e08074b70 2024-05-15T13:20:32.386 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.387+0000 7f6e03fff640 10 client.22192.objecter ms_dispatch 0x7f6e1c000b90 command_reply(tid 1: 0 ) v1 2024-05-15T13:20:32.386 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.387+0000 7f6e03fff640 10 client.22192.objecter _finish_command 1 = system:0 2024-05-15T13:20:32.386 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.387+0000 7f6e03fff640 20 client.22192.objecter put_session s=0x7f6e080744a0 osd=5 4 2024-05-15T13:20:32.386 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.387+0000 7f6e03fff640 15 client.22192.objecter _session_command_op_remove 5 1 2024-05-15T13:20:32.401 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.403+0000 7f6e21bde640 10 client.22192.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-05-15T13:20:32.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.403+0000 7f6e21bde640 15 client.22192.objecter _session_command_op_assign -1 2 2024-05-15T13:20:32.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.403+0000 7f6e21bde640 20 client.22192.objecter _get_session s=0x7f6e080744a0 osd=5 3 2024-05-15T13:20:32.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.403+0000 7f6e21bde640 20 client.22192.objecter put_session s=0x7f6e080744a0 osd=5 3 2024-05-15T13:20:32.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.403+0000 7f6e21bde640 20 client.22192.objecter _get_session s=0x7f6e080744a0 osd=5 3 2024-05-15T13:20:32.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.403+0000 7f6e21bde640 15 client.22192.objecter _session_command_op_remove -1 2 2024-05-15T13:20:32.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.403+0000 7f6e21bde640 20 client.22192.objecter get_session s=0x7f6e080744a0 osd=5 3 2024-05-15T13:20:32.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.403+0000 7f6e21bde640 15 client.22192.objecter _session_command_op_assign 5 2 2024-05-15T13:20:32.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.403+0000 7f6e21bde640 20 client.22192.objecter put_session s=0x7f6e080744a0 osd=5 4 2024-05-15T13:20:32.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.403+0000 7f6e21bde640 10 client.22192.objecter _send_command 2 2024-05-15T13:20:32.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.403+0000 7f6e21bde640 1 -- 172.21.15.117:0/3101673290 --> [v2:172.21.15.178:6816/3713764498,v1:172.21.15.178:6817/3713764498] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f6e1c13bcd0 con 0x7f6e08074b70 2024-05-15T13:20:32.415 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.416+0000 7f6e03fff640 1 -- 172.21.15.117:0/3101673290 <== osd.5 v2:172.21.15.178:6816/3713764498 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f6e1c13bcd0 con 0x7f6e08074b70 2024-05-15T13:20:32.415 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.416+0000 7f6e03fff640 10 client.22192.objecter ms_dispatch 0x7f6e1c000b90 command_reply(tid 2: 0 ) v1 2024-05-15T13:20:32.415 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.416+0000 7f6e03fff640 10 client.22192.objecter _finish_command 2 = system:0 2024-05-15T13:20:32.415 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.416+0000 7f6e03fff640 20 client.22192.objecter put_session s=0x7f6e080744a0 osd=5 4 2024-05-15T13:20:32.416 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.416+0000 7f6e03fff640 15 client.22192.objecter _session_command_op_remove 5 2 2024-05-15T13:20:32.416 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-05-15T13:20:32.416 INFO:teuthology.orchestra.run.smithi117.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-15T13:20:32.416 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-05-15T13:20:32.418 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.417+0000 7f6e21bde640 10 librados: watch_flush enter 2024-05-15T13:20:32.418 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.417+0000 7f6e21bde640 10 librados: watch_flush exit 2024-05-15T13:20:32.418 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.417+0000 7f6e21bde640 10 client.22192.objecter close_session for osd.5 2024-05-15T13:20:32.418 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.417+0000 7f6e21bde640 1 -- 172.21.15.117:0/3101673290 >> [v2:172.21.15.178:6816/3713764498,v1:172.21.15.178:6817/3713764498] conn(0x7f6e08074b70 msgr2=0x7f6e08077010 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:32.418 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.417+0000 7f6e21bde640 1 --2- 172.21.15.117:0/3101673290 >> [v2:172.21.15.178:6816/3713764498,v1:172.21.15.178:6817/3713764498] conn(0x7f6e08074b70 0x7f6e08077010 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:32.418 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.417+0000 7f6e21bde640 20 client.22192.objecter put_session s=0x7f6e080744a0 osd=5 1 2024-05-15T13:20:32.418 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.417+0000 7f6e21bde640 20 client.22192.objecter shutdown clearing up homeless session... 2024-05-15T13:20:32.418 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.417+0000 7f6e21bde640 10 client.22192.objecter successfully canceled tick 2024-05-15T13:20:32.419 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.417+0000 7f6e21bde640 1 -- 172.21.15.117:0/3101673290 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f6e0806ce60 msgr2=0x7f6e0806f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:32.419 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.417+0000 7f6e21bde640 1 --2- 172.21.15.117:0/3101673290 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f6e0806ce60 0x7f6e0806f320 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f6e10007fd0 tx=0x7f6e10009040 comp rx=0 tx=0).stop 2024-05-15T13:20:32.419 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.417+0000 7f6e21bde640 10 monclient: shutdown 2024-05-15T13:20:32.419 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.417+0000 7f6e21bde640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:20:32.419 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.417+0000 7f6e21bde640 1 -- 172.21.15.117:0/3101673290 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6e04003a00 msgr2=0x7f6e1c0b8830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:32.419 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.417+0000 7f6e21bde640 1 --2- 172.21.15.117:0/3101673290 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6e04003a00 0x7f6e1c0b8830 secure :-1 s=READY pgs=5455 cs=0 l=1 rev1=1 crypto rx=0x7f6e0c00c200 tx=0x7f6e1c0ab880 comp rx=0 tx=0).stop 2024-05-15T13:20:32.419 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.417+0000 7f6e1bfff640 1 -- 172.21.15.117:0/3101673290 reap_dead start 2024-05-15T13:20:32.419 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.417+0000 7f6e21bde640 1 -- 172.21.15.117:0/3101673290 shutdown_connections 2024-05-15T13:20:32.419 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.417+0000 7f6e21bde640 1 -- 172.21.15.117:0/3101673290 >> 172.21.15.117:0/3101673290 conn(0x7f6e1c066ed0 msgr2=0x7f6e1c0b30a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:20:32.419 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.417+0000 7f6e21bde640 1 -- 172.21.15.117:0/3101673290 shutdown_connections 2024-05-15T13:20:32.419 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.417+0000 7f6e21bde640 1 -- 172.21.15.117:0/3101673290 wait complete. 2024-05-15T13:20:32.419 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.417+0000 7f6e21bde640 1 librados: shutdown 2024-05-15T13:20:32.429 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-05-15T13:20:32.534 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.535+0000 7ff7fc46a640 10 monclient: get_monmap_and_config 2024-05-15T13:20:32.534 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.535+0000 7ff7fc46a640 10 monclient: build_initial_monmap 2024-05-15T13:20:32.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.535+0000 7ff7fc46a640 10 monclient: monmap: 2024-05-15T13:20:32.535 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T13:20:32.535 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:32.535 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T13:20:32.536693+0000 2024-05-15T13:20:32.535 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T13:20:32.536693+0000 2024-05-15T13:20:32.535 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T13:20:32.535 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:32.535 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T13:20:32.535 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T13:20:32.535 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T13:20:32.535 INFO:teuthology.orchestra.run.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T13:20:32.535 INFO:teuthology.orchestra.run.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T13:20:32.535 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:32.536 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.536+0000 7ff7fc46a640 1 Processor -- start 2024-05-15T13:20:32.536 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.536+0000 7ff7fc46a640 1 -- start start 2024-05-15T13:20:32.536 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.536+0000 7ff7fc46a640 10 monclient: init 2024-05-15T13:20:32.539 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.536+0000 7ff7fc46a640 10 monclient: _reopen_session rank -1 2024-05-15T13:20:32.539 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.536+0000 7ff7fc46a640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-05-15T13:20:32.539 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.537+0000 7ff7fc46a640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff7f40b4d00 0x7ff7f40b1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:32.539 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.537+0000 7ff7fc46a640 10 monclient(hunting): picked mon.noname-b con 0x7ff7f40b4d00 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T13:20:32.539 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.537+0000 7ff7fc46a640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7ff7f40b50e0 0x7ff7f4145c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:32.539 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.537+0000 7ff7fc46a640 10 monclient(hunting): picked mon.noname-a con 0x7ff7f40b50e0 addr v2:172.21.15.117:3300/0 2024-05-15T13:20:32.539 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.537+0000 7ff7fc46a640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7ff7f40a94b0 0x7ff7f40b0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:32.539 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.537+0000 7ff7fc46a640 10 monclient(hunting): picked mon.noname-c con 0x7ff7f40a94b0 addr v2:172.21.15.178:3300/0 2024-05-15T13:20:32.539 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.537+0000 7ff7fc46a640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:32.539 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.537+0000 7ff7fc46a640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7f40b23c0 con 0x7ff7f40b50e0 2024-05-15T13:20:32.540 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.537+0000 7ff7fc46a640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:32.540 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.537+0000 7ff7fc46a640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7f40ab1b0 con 0x7ff7f40a94b0 2024-05-15T13:20:32.540 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.537+0000 7ff7fc46a640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:32.540 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.537+0000 7ff7fc46a640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f4146160 con 0x7ff7f40b4d00 2024-05-15T13:20:32.540 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.537+0000 7ff7fc46a640 10 monclient(hunting): _renew_subs 2024-05-15T13:20:32.540 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.537+0000 7ff7fc46a640 10 monclient(hunting): authenticate will time out at 3595.778868s 2024-05-15T13:20:32.540 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.537+0000 7ff7fa206640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff7f40b4d00 0x7ff7f40b1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:32.540 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.537+0000 7ff7faa07640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7ff7f40a94b0 0x7ff7f40b0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:32.540 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.537+0000 7ff7f9a05640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7ff7f40b50e0 0x7ff7f4145c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:32.540 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.537+0000 7ff7faa07640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7ff7f40a94b0 0x7ff7f40b0fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:37414/0 (socket says 172.21.15.117:37414) 2024-05-15T13:20:32.540 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.537+0000 7ff7faa07640 1 -- 172.21.15.117:0/3861648317 learned_addr learned my addr 172.21.15.117:0/3861648317 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T13:20:32.540 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.537+0000 7ff7faa07640 10 monclient(hunting): get_auth_request con 0x7ff7f40a94b0 auth_method 0 2024-05-15T13:20:32.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.537+0000 7ff7faa07640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:32.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.537+0000 7ff7faa07640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:32.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.537+0000 7ff7faa07640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:32.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.538+0000 7ff7fa206640 10 monclient(hunting): get_auth_request con 0x7ff7f40b4d00 auth_method 0 2024-05-15T13:20:32.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.538+0000 7ff7fa206640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:32.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.538+0000 7ff7fa206640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:32.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.538+0000 7ff7fa206640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:32.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.538+0000 7ff7f9a05640 10 monclient(hunting): get_auth_request con 0x7ff7f40b50e0 auth_method 0 2024-05-15T13:20:32.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.538+0000 7ff7f9a05640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:32.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.538+0000 7ff7f9a05640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:32.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.538+0000 7ff7f9a05640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:32.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.538+0000 7ff7faa07640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:32.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.538+0000 7ff7faa07640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:32.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.538+0000 7ff7faa07640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:32.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.538+0000 7ff7fa206640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:32.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.538+0000 7ff7fa206640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:32.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.538+0000 7ff7fa206640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:32.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.538+0000 7ff7f9a05640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:32.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.538+0000 7ff7f9a05640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:32.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.538+0000 7ff7f9a05640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:32.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.539+0000 7ff7faa07640 10 monclient(hunting): handle_auth_done global_id 22195 payload 274 2024-05-15T13:20:32.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.540+0000 7ff7faa07640 1 -- 172.21.15.117:0/3861648317 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff7f40b4d00 msgr2=0x7ff7f40b1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T13:20:32.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.540+0000 7ff7faa07640 1 --2- 172.21.15.117:0/3861648317 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff7f40b4d00 0x7ff7f40b1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:32.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.540+0000 7ff7faa07640 1 -- 172.21.15.117:0/3861648317 >> v2:172.21.15.117:3300/0 conn(0x7ff7f40b50e0 msgr2=0x7ff7f4145c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T13:20:32.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.540+0000 7ff7faa07640 1 --2- 172.21.15.117:0/3861648317 >> v2:172.21.15.117:3300/0 conn(0x7ff7f40b50e0 0x7ff7f4145c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:32.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.540+0000 7ff7faa07640 10 monclient: _finish_hunting 0 2024-05-15T13:20:32.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.540+0000 7ff7faa07640 1 monclient: found mon.noname-c 2024-05-15T13:20:32.543 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.540+0000 7ff7faa07640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:20:32.543 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.540+0000 7ff7faa07640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.178:3300/0 2024-05-15T13:20:32.543 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.540+0000 7ff7faa07640 1 -- 172.21.15.117:0/3861648317 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7f40b1520 con 0x7ff7f40a94b0 2024-05-15T13:20:32.543 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.542+0000 7ff7faa07640 1 --2- 172.21.15.117:0/3861648317 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff7f40a94b0 0x7ff7f40b0fe0 secure :-1 s=READY pgs=5456 cs=0 l=1 rev1=1 crypto rx=0x7ff7f40b5680 tx=0x7ff7ec02be40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1ee34a9d3d20170 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:32.544 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.546+0000 7ff7f9204640 1 -- 172.21.15.117:0/3861648317 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7ec02c9e0 con 0x7ff7f40a94b0 2024-05-15T13:20:32.544 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.546+0000 7ff7f9204640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:32.545 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.546+0000 7ff7f9204640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T13:20:32.545 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.546+0000 7ff7f9204640 10 monclient: dump: 2024-05-15T13:20:32.545 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:32.546 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:32.546 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:32.546 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:32.546 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:32.546 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:32.546 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:32.546 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:32.546 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:32.546 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:32.546 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.546+0000 7ff7f9204640 10 monclient: _finish_auth 0 2024-05-15T13:20:32.547 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.546+0000 7ff7f9204640 10 monclient: _check_auth_tickets 2024-05-15T13:20:32.547 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.546+0000 7ff7f9204640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:20:32.547 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.546+0000 7ff7f9204640 1 -- 172.21.15.117:0/3861648317 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7ec02cb80 con 0x7ff7f40a94b0 2024-05-15T13:20:32.547 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.546+0000 7ff7fc46a640 5 monclient: authenticate success, global_id 22195 2024-05-15T13:20:32.547 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.546+0000 7ff7fc46a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T13:20:32.547 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.546+0000 7ff7f9204640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:20:32.547 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.546+0000 7ff7fc46a640 10 monclient: get_monmap_and_config success 2024-05-15T13:20:32.547 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.546+0000 7ff7f9204640 1 -- 172.21.15.117:0/3861648317 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7ec0343f0 con 0x7ff7f40a94b0 2024-05-15T13:20:32.547 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.546+0000 7ff7f9204640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:32.547 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.546+0000 7ff7fc46a640 4 set_mon_vals no callback set 2024-05-15T13:20:32.547 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.546+0000 7ff7f9204640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T13:20:32.547 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.546+0000 7ff7f9204640 10 monclient: dump: 2024-05-15T13:20:32.547 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:32.547 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:32.548 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:32.548 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:32.548 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:32.548 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:32.548 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:32.548 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:32.548 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:32.548 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:32.552 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.546+0000 7ff7fc46a640 10 monclient: shutdown 2024-05-15T13:20:32.552 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.546+0000 7ff7fc46a640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:20:32.552 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.546+0000 7ff7fc46a640 1 -- 172.21.15.117:0/3861648317 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff7f40a94b0 msgr2=0x7ff7f40b0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:32.552 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.546+0000 7ff7fc46a640 1 --2- 172.21.15.117:0/3861648317 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff7f40a94b0 0x7ff7f40b0fe0 secure :-1 s=READY pgs=5456 cs=0 l=1 rev1=1 crypto rx=0x7ff7f40b5680 tx=0x7ff7ec02be40 comp rx=0 tx=0).stop 2024-05-15T13:20:32.552 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.546+0000 7ff7fc46a640 1 -- 172.21.15.117:0/3861648317 shutdown_connections 2024-05-15T13:20:32.552 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.546+0000 7ff7fc46a640 1 --2- 172.21.15.117:0/3861648317 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff7f40a94b0 0x7ff7f40b0fe0 unknown :-1 s=CLOSED pgs=5456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:32.552 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.546+0000 7ff7fc46a640 1 --2- 172.21.15.117:0/3861648317 >> v2:172.21.15.117:3300/0 conn(0x7ff7f40b50e0 0x7ff7f4145c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:32.552 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.546+0000 7ff7fc46a640 1 --2- 172.21.15.117:0/3861648317 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff7f40b4d00 0x7ff7f40b1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:32.552 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.546+0000 7ff7fc46a640 1 -- 172.21.15.117:0/3861648317 >> 172.21.15.117:0/3861648317 conn(0x7ff7f4066cc0 msgr2=0x7ff7f40670d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:20:32.553 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.546+0000 7ff7fc46a640 1 -- 172.21.15.117:0/3861648317 shutdown_connections 2024-05-15T13:20:32.553 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.546+0000 7ff7fc46a640 1 -- 172.21.15.117:0/3861648317 wait complete. 2024-05-15T13:20:32.553 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.547+0000 7ff7fc46a640 10 monclient: build_initial_monmap 2024-05-15T13:20:32.553 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.547+0000 7ff7fc46a640 10 monclient: monmap: 2024-05-15T13:20:32.553 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T13:20:32.553 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T13:20:32.553 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-05-15T13:20:32.553 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-05-15T13:20:32.553 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T13:20:32.553 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:32.553 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T13:20:32.553 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T13:20:32.553 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T13:20:32.553 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:32.554 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.547+0000 7ff7fc46a640 1 librados: starting msgr at 2024-05-15T13:20:32.554 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.547+0000 7ff7fc46a640 1 librados: starting objecter 2024-05-15T13:20:32.554 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.548+0000 7ff7fc46a640 1 Processor -- start 2024-05-15T13:20:32.554 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.548+0000 7ff7fc46a640 1 -- start start 2024-05-15T13:20:32.554 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.548+0000 7ff7fc46a640 1 librados: setting wanted keys 2024-05-15T13:20:32.554 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.548+0000 7ff7fc46a640 1 librados: calling monclient init 2024-05-15T13:20:32.554 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.548+0000 7ff7fc46a640 10 monclient: init 2024-05-15T13:20:32.554 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.548+0000 7ff7fc46a640 10 monclient: _reopen_session rank -1 2024-05-15T13:20:32.554 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.548+0000 7ff7fc46a640 10 monclient: _add_conns ranks=[2,0,1] 2024-05-15T13:20:32.554 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.548+0000 7ff7fc46a640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff7f40a94b0 0x7ff7f4145650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:32.554 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.548+0000 7ff7fc46a640 10 monclient(hunting): picked mon.noname-c con 0x7ff7f40a94b0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T13:20:32.555 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.548+0000 7ff7fc46a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff7f40b4d00 0x7ff7f41d7f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:32.555 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.548+0000 7ff7fc46a640 10 monclient(hunting): picked mon.noname-a con 0x7ff7f40b4d00 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T13:20:32.555 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.548+0000 7ff7fc46a640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff7f40b0090 0x7ff7f41dbc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:32.555 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.548+0000 7ff7fc46a640 10 monclient(hunting): picked mon.noname-b con 0x7ff7f40b0090 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T13:20:32.555 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.548+0000 7ff7fc46a640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:32.555 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.548+0000 7ff7fc46a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f4142700 con 0x7ff7f40b4d00 2024-05-15T13:20:32.555 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.548+0000 7ff7fc46a640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:32.555 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.548+0000 7ff7fc46a640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f40a9b50 con 0x7ff7f40b0090 2024-05-15T13:20:32.555 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.548+0000 7ff7fc46a640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:32.555 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.548+0000 7ff7fc46a640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f4142880 con 0x7ff7f40a94b0 2024-05-15T13:20:32.555 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.548+0000 7ff7fc46a640 10 monclient(hunting): _renew_subs 2024-05-15T13:20:32.555 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.548+0000 7ff7fc46a640 10 monclient(hunting): authenticate will time out at 3595.790428s 2024-05-15T13:20:32.555 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.549+0000 7ff7f9a05640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff7f40b4d00 0x7ff7f41d7f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:32.555 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.549+0000 7ff7fa206640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff7f40a94b0 0x7ff7f4145650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:32.556 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.549+0000 7ff7faa07640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff7f40b0090 0x7ff7f41dbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:32.556 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.549+0000 7ff7f9a05640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff7f40b4d00 0x7ff7f41d7f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:44308/0 (socket says 172.21.15.117:44308) 2024-05-15T13:20:32.556 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.549+0000 7ff7f9a05640 1 -- 172.21.15.117:0/937619229 learned_addr learned my addr 172.21.15.117:0/937619229 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T13:20:32.556 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.549+0000 7ff7f9a05640 10 monclient(hunting): get_auth_request con 0x7ff7f40b4d00 auth_method 0 2024-05-15T13:20:32.556 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.549+0000 7ff7f9a05640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:32.556 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.549+0000 7ff7f9a05640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:32.556 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.549+0000 7ff7f9a05640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:32.556 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.549+0000 7ff7fa206640 10 monclient(hunting): get_auth_request con 0x7ff7f40a94b0 auth_method 0 2024-05-15T13:20:32.556 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.549+0000 7ff7fa206640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:32.556 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.549+0000 7ff7fa206640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:32.556 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.549+0000 7ff7fa206640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:32.556 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.549+0000 7ff7faa07640 10 monclient(hunting): get_auth_request con 0x7ff7f40b0090 auth_method 0 2024-05-15T13:20:32.556 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.549+0000 7ff7faa07640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:32.556 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.549+0000 7ff7faa07640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:32.556 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.549+0000 7ff7faa07640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:32.556 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.549+0000 7ff7f9a05640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:32.556 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.549+0000 7ff7f9a05640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:32.557 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.549+0000 7ff7f9a05640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:32.557 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.550+0000 7ff7faa07640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:32.557 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.550+0000 7ff7faa07640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:32.557 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.550+0000 7ff7faa07640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:32.557 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.550+0000 7ff7fa206640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:32.557 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.550+0000 7ff7fa206640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:32.557 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.550+0000 7ff7fa206640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:32.557 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.552+0000 7ff7f9a05640 10 monclient(hunting): handle_auth_done global_id 16080 payload 824 2024-05-15T13:20:32.557 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.552+0000 7ff7f9a05640 1 -- 172.21.15.117:0/937619229 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff7f40a94b0 msgr2=0x7ff7f4145650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:32.557 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.552+0000 7ff7f9a05640 1 --2- 172.21.15.117:0/937619229 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff7f40a94b0 0x7ff7f4145650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:32.557 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.552+0000 7ff7f9a05640 1 -- 172.21.15.117:0/937619229 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff7f40b0090 msgr2=0x7ff7f41dbc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:32.557 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.552+0000 7ff7f9a05640 1 --2- 172.21.15.117:0/937619229 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff7f40b0090 0x7ff7f41dbc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:32.557 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.552+0000 7ff7f9a05640 10 monclient: _finish_hunting 0 2024-05-15T13:20:32.558 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.552+0000 7ff7f9a05640 1 monclient: found mon.noname-a 2024-05-15T13:20:32.558 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.552+0000 7ff7f9a05640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:20:32.558 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.552+0000 7ff7f9a05640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.117:3300/0 2024-05-15T13:20:32.558 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.552+0000 7ff7f9a05640 1 -- 172.21.15.117:0/937619229 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7f41db740 con 0x7ff7f40b4d00 2024-05-15T13:20:32.558 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.552+0000 7ff7faa07640 1 --2- 172.21.15.117:0/937619229 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff7f40b0090 0x7ff7f41dbc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-15T13:20:32.559 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.555+0000 7ff7f9a05640 1 --2- 172.21.15.117:0/937619229 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff7f40b4d00 0x7ff7f41d7f10 secure :-1 s=READY pgs=3305 cs=0 l=1 rev1=1 crypto rx=0x7ff7e400e8a0 tx=0x7ff7e400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:32.560 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.555+0000 7ff7eaffd640 10 client.?.objecter ms_handle_connect 0x7ff7f40b4d00 2024-05-15T13:20:32.560 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.555+0000 7ff7eaffd640 10 client.?.objecter resend_mon_ops 2024-05-15T13:20:32.560 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.559+0000 7ff7eaffd640 1 -- 172.21.15.117:0/937619229 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7e4017070 con 0x7ff7f40b4d00 2024-05-15T13:20:32.560 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.559+0000 7ff7eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:32.560 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.559+0000 7ff7eaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T13:20:32.560 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.559+0000 7ff7eaffd640 10 monclient: dump: 2024-05-15T13:20:32.560 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:32.560 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:32.560 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:32.560 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:32.560 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:32.560 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:32.561 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:32.561 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:32.561 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:32.561 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:32.561 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.559+0000 7ff7eaffd640 10 monclient: _finish_auth 0 2024-05-15T13:20:32.561 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.559+0000 7ff7eaffd640 10 monclient: _check_auth_tickets 2024-05-15T13:20:32.561 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.559+0000 7ff7eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:20:32.561 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.559+0000 7ff7eaffd640 1 -- 172.21.15.117:0/937619229 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7e400c940 con 0x7ff7f40b4d00 2024-05-15T13:20:32.561 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.559+0000 7ff7fc46a640 5 monclient: authenticate success, global_id 16080 2024-05-15T13:20:32.561 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.559+0000 7ff7fc46a640 10 monclient: _renew_subs 2024-05-15T13:20:32.561 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.559+0000 7ff7fc46a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T13:20:32.561 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.559+0000 7ff7fc46a640 1 -- 172.21.15.117:0/937619229 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7f40b0b30 con 0x7ff7f40b4d00 2024-05-15T13:20:32.561 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.559+0000 7ff7eaffd640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:20:32.562 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.559+0000 7ff7eaffd640 1 -- 172.21.15.117:0/937619229 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7e400cd30 con 0x7ff7f40b4d00 2024-05-15T13:20:32.562 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.559+0000 7ff7eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:32.562 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.559+0000 7ff7eaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T13:20:32.562 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.559+0000 7ff7eaffd640 10 monclient: dump: 2024-05-15T13:20:32.562 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:32.562 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:32.562 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:32.562 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:32.562 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:32.562 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:32.562 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:32.562 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:32.562 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:32.562 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:32.563 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.559+0000 7ff7f8a03640 4 set_mon_vals no callback set 2024-05-15T13:20:32.563 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.559+0000 7ff7fc46a640 10 client.16080.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:20:32.563 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.559+0000 7ff7fc46a640 10 monclient: _renew_subs 2024-05-15T13:20:32.563 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.559+0000 7ff7fc46a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T13:20:32.563 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.559+0000 7ff7fc46a640 1 -- 172.21.15.117:0/937619229 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7f41dc180 con 0x7ff7f40b4d00 2024-05-15T13:20:32.563 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.559+0000 7ff7fc46a640 1 librados: init done 2024-05-15T13:20:32.563 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.560+0000 7ff7fc46a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T13:20:32.563 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.560+0000 7ff7fc46a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:20:32.563 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.560+0000 7ff7fc46a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T13:20:32.563 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.560+0000 7ff7fc46a640 1 -- 172.21.15.117:0/937619229 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7f40ab8e0 con 0x7ff7f40b4d00 2024-05-15T13:20:32.588 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.589+0000 7ff7eaffd640 1 -- 172.21.15.117:0/937619229 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7ff7e401a070 con 0x7ff7f40b4d00 2024-05-15T13:20:32.588 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.589+0000 7ff7eaffd640 1 --2- 172.21.15.117:0/937619229 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7ff7d806ceb0 0x7ff7d806f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:32.588 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.589+0000 7ff7fa206640 1 --2- 172.21.15.117:0/937619229 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7ff7d806ceb0 0x7ff7d806f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:32.589 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.589+0000 7ff7fa206640 10 monclient: get_auth_request con 0x7ff7d806ceb0 auth_method 0 2024-05-15T13:20:32.589 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.590+0000 7ff7fa206640 1 --2- 172.21.15.117:0/937619229 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7ff7d806ceb0 0x7ff7d806f370 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7ff7f0009b40 tx=0x7ff7f0009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:32.589 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.590+0000 7ff7eaffd640 10 client.16080.objecter ms_handle_connect 0x7ff7d806ceb0 2024-05-15T13:20:32.593 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.594+0000 7ff7eaffd640 1 -- 172.21.15.117:0/937619229 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(1225..1225 src has 912..1225) v4 ==== 6365+0+0 (secure 0 0 0) 0x7ff7e4057010 con 0x7ff7f40b4d00 2024-05-15T13:20:32.593 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.594+0000 7ff7eaffd640 10 client.16080.objecter ms_dispatch 0x7ff7f4000b90 osd_map(1225..1225 src has 912..1225) v4 2024-05-15T13:20:32.593 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.594+0000 7ff7eaffd640 3 client.16080.objecter handle_osd_map got epochs [1225,1225] > 0 2024-05-15T13:20:32.593 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.594+0000 7ff7eaffd640 3 client.16080.objecter handle_osd_map decoding full epoch 1225 2024-05-15T13:20:32.593 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.594+0000 7ff7eaffd640 20 client.16080.objecter dump_active .. 0 homeless 2024-05-15T13:20:32.879 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.880+0000 7ff7eaffd640 1 -- 172.21.15.117:0/937619229 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7ff7e4058f70 con 0x7ff7f40b4d00 2024-05-15T13:20:32.879 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.880+0000 7ff7eaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:20:32.879 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:32.880+0000 7ff7eaffd640 10 monclient: _finish_command 1 = system:0 2024-05-15T13:20:33.064 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.065+0000 7ff7fc46a640 1 -- 172.21.15.117:0/937619229 --> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7ff7f404f0e0 con 0x7ff7d806ceb0 2024-05-15T13:20:33.064 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.065+0000 7ff7eaffd640 1 -- 172.21.15.117:0/937619229 <== mgr.4107 v2:172.21.15.117:6800/1785766523 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7ff7f404f0e0 con 0x7ff7d806ceb0 2024-05-15T13:20:33.064 INFO:teuthology.orchestra.run.smithi117.stderr:instructed osd(s) 5 to deep-scrub 2024-05-15T13:20:33.068 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.068+0000 7ff7fc46a640 10 librados: watch_flush enter 2024-05-15T13:20:33.068 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.068+0000 7ff7fc46a640 10 librados: watch_flush exit 2024-05-15T13:20:33.068 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.068+0000 7ff7fc46a640 20 client.16080.objecter shutdown clearing up homeless session... 2024-05-15T13:20:33.068 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.068+0000 7ff7fc46a640 10 client.16080.objecter successfully canceled tick 2024-05-15T13:20:33.068 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.068+0000 7ff7fc46a640 1 -- 172.21.15.117:0/937619229 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7ff7d806ceb0 msgr2=0x7ff7d806f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:33.068 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.068+0000 7ff7fc46a640 1 --2- 172.21.15.117:0/937619229 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7ff7d806ceb0 0x7ff7d806f370 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7ff7f0009b40 tx=0x7ff7f0009190 comp rx=0 tx=0).stop 2024-05-15T13:20:33.068 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.068+0000 7ff7fc46a640 10 monclient: shutdown 2024-05-15T13:20:33.069 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.068+0000 7ff7fc46a640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:20:33.069 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.068+0000 7ff7fc46a640 1 -- 172.21.15.117:0/937619229 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff7f40b4d00 msgr2=0x7ff7f41d7f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:33.069 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.068+0000 7ff7fc46a640 1 --2- 172.21.15.117:0/937619229 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff7f40b4d00 0x7ff7f41d7f10 secure :-1 s=READY pgs=3305 cs=0 l=1 rev1=1 crypto rx=0x7ff7e400e8a0 tx=0x7ff7e400ed60 comp rx=0 tx=0).stop 2024-05-15T13:20:33.069 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.069+0000 7ff7fc46a640 1 -- 172.21.15.117:0/937619229 shutdown_connections 2024-05-15T13:20:33.069 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.069+0000 7ff7fc46a640 1 --2- 172.21.15.117:0/937619229 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7ff7d806ceb0 0x7ff7d806f370 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:33.069 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.069+0000 7ff7fc46a640 1 --2- 172.21.15.117:0/937619229 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff7f40b0090 0x7ff7f41dbc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:33.069 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.069+0000 7ff7fc46a640 1 --2- 172.21.15.117:0/937619229 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff7f40b4d00 0x7ff7f41d7f10 unknown :-1 s=CLOSED pgs=3305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:33.069 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.069+0000 7ff7fc46a640 1 --2- 172.21.15.117:0/937619229 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff7f40a94b0 0x7ff7f4145650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:33.069 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.069+0000 7ff7fc46a640 1 -- 172.21.15.117:0/937619229 >> 172.21.15.117:0/937619229 conn(0x7ff7f4066cc0 msgr2=0x7ff7f40b6f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:20:33.069 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.069+0000 7ff7fc46a640 1 -- 172.21.15.117:0/937619229 shutdown_connections 2024-05-15T13:20:33.069 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.069+0000 7ff7fc46a640 1 -- 172.21.15.117:0/937619229 wait complete. 2024-05-15T13:20:33.069 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.069+0000 7ff7fc46a640 1 librados: shutdown 2024-05-15T13:20:33.081 INFO:tasks.ceph:Scrubbing osd.6 2024-05-15T13:20:33.081 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-05-15T13:20:33.185 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.186+0000 7ff00c7de640 10 monclient: get_monmap_and_config 2024-05-15T13:20:33.185 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.186+0000 7ff00c7de640 10 monclient: build_initial_monmap 2024-05-15T13:20:33.185 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.186+0000 7ff00c7de640 10 monclient: monmap: 2024-05-15T13:20:33.185 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T13:20:33.185 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:33.185 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T13:20:33.187266+0000 2024-05-15T13:20:33.185 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T13:20:33.187266+0000 2024-05-15T13:20:33.186 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T13:20:33.186 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:33.186 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T13:20:33.186 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T13:20:33.186 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T13:20:33.186 INFO:teuthology.orchestra.run.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T13:20:33.186 INFO:teuthology.orchestra.run.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T13:20:33.186 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:33.186 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.186+0000 7ff00c7de640 1 Processor -- start 2024-05-15T13:20:33.186 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.186+0000 7ff00c7de640 1 -- start start 2024-05-15T13:20:33.186 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.186+0000 7ff00c7de640 10 monclient: init 2024-05-15T13:20:33.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.187+0000 7ff00c7de640 10 monclient: _reopen_session rank -1 2024-05-15T13:20:33.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.187+0000 7ff00c7de640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-05-15T13:20:33.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.187+0000 7ff00c7de640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff0040b5180 addr v1:172.21.15.117:6789/0 2024-05-15T13:20:33.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.187+0000 7ff00c7de640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff00413ff40 0x7ff0040a9980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:33.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.187+0000 7ff00c7de640 10 monclient(hunting): picked mon.noname-b con 0x7ff00413ff40 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T13:20:33.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.187+0000 7ff00c7de640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff004142680 addr v1:172.21.15.178:6789/0 2024-05-15T13:20:33.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.187+0000 7ff00c7de640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0040aa530 con 0x7ff0040b5180 2024-05-15T13:20:33.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.187+0000 7ff00c7de640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0040ab6b0 con 0x7ff004142680 2024-05-15T13:20:33.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.187+0000 7ff00c7de640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:33.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.187+0000 7ff00c7de640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0040b5560 con 0x7ff00413ff40 2024-05-15T13:20:33.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.187+0000 7ff00c7de640 10 monclient(hunting): _renew_subs 2024-05-15T13:20:33.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.187+0000 7ff00c7de640 10 monclient(hunting): authenticate will time out at 3596.429551s 2024-05-15T13:20:33.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.188+0000 7ff00ad7b640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7ff004142680 0x7ff004142b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.117:36852/0 (socket says 172.21.15.117:36852) 2024-05-15T13:20:33.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.188+0000 7ff00a57a640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7ff0040b5180 0x7ff00413f830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:41492/0 (socket says 172.21.15.117:41492) 2024-05-15T13:20:33.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.188+0000 7ff00ad7b640 1 -- 172.21.15.117:0/1022421716 learned_addr learned my addr 172.21.15.117:0/1022421716 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-05-15T13:20:33.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.188+0000 7ff009d79640 1 --2- 172.21.15.117:0/1022421716 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff00413ff40 0x7ff0040a9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:33.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.188+0000 7ff009d79640 10 monclient(hunting): get_auth_request con 0x7ff00413ff40 auth_method 0 2024-05-15T13:20:33.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.188+0000 7ff009d79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:33.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.188+0000 7ff009d79640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:33.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.188+0000 7ff009d79640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:33.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.189+0000 7ff009d79640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:33.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.189+0000 7ff009d79640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:33.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.189+0000 7ff009d79640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:33.190 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.190+0000 7ff009578640 1 -- 172.21.15.117:0/1022421716 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2381321098 0 0) 0x7ff0040ab6b0 con 0x7ff004142680 2024-05-15T13:20:33.190 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.190+0000 7ff009578640 10 monclient(hunting): _negotiate 2024-05-15T13:20:33.190 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.190+0000 7ff009578640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:33.190 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.190+0000 7ff009578640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:33.191 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.190+0000 7ff009578640 10 monclient(hunting): my global_id is 22201 2024-05-15T13:20:33.191 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.190+0000 7ff009578640 1 -- 172.21.15.117:0/1022421716 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feff0003520 con 0x7ff004142680 2024-05-15T13:20:33.191 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.191+0000 7ff009578640 1 -- 172.21.15.117:0/1022421716 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 457631890 0 0) 0x7ff0040aa530 con 0x7ff0040b5180 2024-05-15T13:20:33.191 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.191+0000 7ff009578640 10 monclient(hunting): _negotiate 2024-05-15T13:20:33.191 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.191+0000 7ff009578640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:33.191 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.191+0000 7ff009578640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:33.191 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.191+0000 7ff009578640 10 monclient(hunting): my global_id is 16083 2024-05-15T13:20:33.191 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.191+0000 7ff009578640 1 -- 172.21.15.117:0/1022421716 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0040ab6b0 con 0x7ff0040b5180 2024-05-15T13:20:33.191 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.191+0000 7ff009d79640 10 monclient(hunting): handle_auth_done global_id 18770 payload 274 2024-05-15T13:20:33.191 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.191+0000 7ff009d79640 1 -- 172.21.15.117:0/1022421716 >> v1:172.21.15.178:6789/0 conn(0x7ff004142680 legacy=0x7ff004142b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:33.191 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.191+0000 7ff009d79640 1 -- 172.21.15.117:0/1022421716 >> v1:172.21.15.117:6789/0 conn(0x7ff0040b5180 legacy=0x7ff00413f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:33.191 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.191+0000 7ff009d79640 10 monclient: _finish_hunting 0 2024-05-15T13:20:33.191 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.191+0000 7ff009d79640 1 monclient: found mon.noname-b 2024-05-15T13:20:33.191 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.191+0000 7ff009d79640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:20:33.191 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.191+0000 7ff009d79640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.117:3301/0 2024-05-15T13:20:33.192 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.191+0000 7ff009d79640 1 -- 172.21.15.117:0/1022421716 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0040aa010 con 0x7ff00413ff40 2024-05-15T13:20:33.192 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.193+0000 7ff009d79640 1 --2- 172.21.15.117:0/1022421716 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff00413ff40 0x7ff0040a9980 secure :-1 s=READY pgs=3387 cs=0 l=1 rev1=1 crypto rx=0x7feff40097d0 tx=0x7feff402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aead388b2204398b server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:33.195 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.197+0000 7ff009578640 1 -- 172.21.15.117:0/1022421716 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff4035020 con 0x7ff00413ff40 2024-05-15T13:20:33.196 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.197+0000 7ff009578640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:33.196 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.197+0000 7ff009578640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T13:20:33.196 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.197+0000 7ff009578640 10 monclient: dump: 2024-05-15T13:20:33.196 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:33.196 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:33.196 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:33.196 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:33.196 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:33.196 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:33.196 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:33.196 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:33.196 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:33.196 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:33.196 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.197+0000 7ff009578640 10 monclient: _finish_auth 0 2024-05-15T13:20:33.197 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.197+0000 7ff009578640 10 monclient: _check_auth_tickets 2024-05-15T13:20:33.197 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.197+0000 7ff009578640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:20:33.197 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.197+0000 7ff009578640 1 -- 172.21.15.117:0/1022421716 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feff4031800 con 0x7ff00413ff40 2024-05-15T13:20:33.197 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.197+0000 7ff00c7de640 5 monclient: authenticate success, global_id 18770 2024-05-15T13:20:33.197 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.197+0000 7ff00c7de640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T13:20:33.197 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.197+0000 7ff009578640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:20:33.197 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.197+0000 7ff00c7de640 10 monclient: get_monmap_and_config success 2024-05-15T13:20:33.197 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.197+0000 7ff009578640 1 -- 172.21.15.117:0/1022421716 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff4031bd0 con 0x7ff00413ff40 2024-05-15T13:20:33.197 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.197+0000 7ff00c7de640 4 set_mon_vals no callback set 2024-05-15T13:20:33.197 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.197+0000 7ff009578640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:33.197 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.197+0000 7ff009578640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T13:20:33.197 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.197+0000 7ff009578640 10 monclient: dump: 2024-05-15T13:20:33.197 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:33.197 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:33.197 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:33.198 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:33.198 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:33.198 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:33.198 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:33.198 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:33.198 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:33.198 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:33.201 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.197+0000 7ff00c7de640 10 monclient: shutdown 2024-05-15T13:20:33.201 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.197+0000 7ff00c7de640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:20:33.201 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.197+0000 7ff00c7de640 1 -- 172.21.15.117:0/1022421716 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff00413ff40 msgr2=0x7ff0040a9980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:33.201 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.197+0000 7ff00c7de640 1 --2- 172.21.15.117:0/1022421716 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff00413ff40 0x7ff0040a9980 secure :-1 s=READY pgs=3387 cs=0 l=1 rev1=1 crypto rx=0x7feff40097d0 tx=0x7feff402b8a0 comp rx=0 tx=0).stop 2024-05-15T13:20:33.201 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.198+0000 7ff00c7de640 1 -- 172.21.15.117:0/1022421716 shutdown_connections 2024-05-15T13:20:33.201 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.198+0000 7ff00c7de640 1 --2- 172.21.15.117:0/1022421716 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff00413ff40 0x7ff0040a9980 unknown :-1 s=CLOSED pgs=3387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:33.202 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.198+0000 7ff00c7de640 1 -- 172.21.15.117:0/1022421716 >> 172.21.15.117:0/1022421716 conn(0x7ff004066ed0 msgr2=0x7ff0040672e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:20:33.202 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.198+0000 7ff00c7de640 1 -- 172.21.15.117:0/1022421716 shutdown_connections 2024-05-15T13:20:33.202 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.198+0000 7ff00c7de640 1 -- 172.21.15.117:0/1022421716 wait complete. 2024-05-15T13:20:33.202 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.198+0000 7ff00c7de640 10 monclient: build_initial_monmap 2024-05-15T13:20:33.202 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.198+0000 7ff00c7de640 10 monclient: monmap: 2024-05-15T13:20:33.202 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T13:20:33.202 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T13:20:33.202 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-05-15T13:20:33.202 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-05-15T13:20:33.202 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T13:20:33.202 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:33.202 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T13:20:33.202 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T13:20:33.202 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T13:20:33.203 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:33.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.199+0000 7ff00c7de640 1 librados: starting msgr at 2024-05-15T13:20:33.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.199+0000 7ff00c7de640 1 librados: starting objecter 2024-05-15T13:20:33.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.199+0000 7ff00c7de640 1 Processor -- start 2024-05-15T13:20:33.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.199+0000 7ff00c7de640 1 -- start start 2024-05-15T13:20:33.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.199+0000 7ff00c7de640 1 librados: setting wanted keys 2024-05-15T13:20:33.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.199+0000 7ff00c7de640 1 librados: calling monclient init 2024-05-15T13:20:33.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.199+0000 7ff00c7de640 10 monclient: init 2024-05-15T13:20:33.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.199+0000 7ff00c7de640 10 monclient: _reopen_session rank -1 2024-05-15T13:20:33.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.199+0000 7ff00c7de640 10 monclient: _add_conns ranks=[0,1,2] 2024-05-15T13:20:33.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.199+0000 7ff00c7de640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff0040b5180 0x7ff0040ac9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:33.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.199+0000 7ff00c7de640 10 monclient(hunting): picked mon.noname-a con 0x7ff0040b5180 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T13:20:33.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.199+0000 7ff00c7de640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff00413ff40 0x7ff0041362a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:33.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.199+0000 7ff00c7de640 10 monclient(hunting): picked mon.noname-b con 0x7ff00413ff40 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T13:20:33.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.199+0000 7ff00c7de640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff004142680 0x7ff004137e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:33.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.199+0000 7ff00c7de640 10 monclient(hunting): picked mon.noname-c con 0x7ff004142680 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T13:20:33.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.199+0000 7ff00c7de640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:33.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.199+0000 7ff00c7de640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff00414e100 con 0x7ff0040b5180 2024-05-15T13:20:33.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.199+0000 7ff00c7de640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:33.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.199+0000 7ff00c7de640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff00414df80 con 0x7ff00413ff40 2024-05-15T13:20:33.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.199+0000 7ff00c7de640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:33.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.199+0000 7ff00c7de640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff00414e280 con 0x7ff004142680 2024-05-15T13:20:33.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.199+0000 7ff00c7de640 10 monclient(hunting): _renew_subs 2024-05-15T13:20:33.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.199+0000 7ff00c7de640 10 monclient(hunting): authenticate will time out at 3596.441576s 2024-05-15T13:20:33.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.200+0000 7ff00ad7b640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff004142680 0x7ff004137e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:33.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.200+0000 7ff00a57a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff0040b5180 0x7ff0040ac9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:33.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.200+0000 7ff009d79640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff00413ff40 0x7ff0041362a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:33.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.200+0000 7ff00a57a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff0040b5180 0x7ff0040ac9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:44324/0 (socket says 172.21.15.117:44324) 2024-05-15T13:20:33.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.200+0000 7ff00a57a640 1 -- 172.21.15.117:0/3872554962 learned_addr learned my addr 172.21.15.117:0/3872554962 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T13:20:33.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.200+0000 7ff00a57a640 10 monclient(hunting): get_auth_request con 0x7ff0040b5180 auth_method 0 2024-05-15T13:20:33.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.200+0000 7ff00a57a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:33.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.200+0000 7ff00a57a640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:33.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.200+0000 7ff00a57a640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:33.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.200+0000 7ff009d79640 10 monclient(hunting): get_auth_request con 0x7ff00413ff40 auth_method 0 2024-05-15T13:20:33.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.200+0000 7ff009d79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:33.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.200+0000 7ff009d79640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:33.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.200+0000 7ff009d79640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:33.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.200+0000 7ff00ad7b640 10 monclient(hunting): get_auth_request con 0x7ff004142680 auth_method 0 2024-05-15T13:20:33.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.200+0000 7ff00ad7b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:33.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.200+0000 7ff00ad7b640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:33.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.200+0000 7ff00ad7b640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:33.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.201+0000 7ff009d79640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:33.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.201+0000 7ff009d79640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:33.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.201+0000 7ff009d79640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:33.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.201+0000 7ff00a57a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:33.206 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.201+0000 7ff00a57a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:33.206 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.201+0000 7ff00a57a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:33.206 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.201+0000 7ff00ad7b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:33.206 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.201+0000 7ff00ad7b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:33.206 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.201+0000 7ff00ad7b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:33.206 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.203+0000 7ff009d79640 10 monclient(hunting): handle_auth_done global_id 22204 payload 824 2024-05-15T13:20:33.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.203+0000 7ff009d79640 1 -- 172.21.15.117:0/3872554962 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff004142680 msgr2=0x7ff004137e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:33.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.203+0000 7ff009d79640 1 --2- 172.21.15.117:0/3872554962 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff004142680 0x7ff004137e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:33.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.203+0000 7ff009d79640 1 -- 172.21.15.117:0/3872554962 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff0040b5180 msgr2=0x7ff0040ac9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:33.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.203+0000 7ff009d79640 1 --2- 172.21.15.117:0/3872554962 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff0040b5180 0x7ff0040ac9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:33.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.203+0000 7ff009d79640 10 monclient: _finish_hunting 0 2024-05-15T13:20:33.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.203+0000 7ff009d79640 1 monclient: found mon.noname-b 2024-05-15T13:20:33.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.203+0000 7ff009d79640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:20:33.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.203+0000 7ff009d79640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2024-05-15T13:20:33.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.203+0000 7ff009d79640 1 -- 172.21.15.117:0/3872554962 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0040abef0 con 0x7ff00413ff40 2024-05-15T13:20:33.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.206+0000 7ff009d79640 1 --2- 172.21.15.117:0/3872554962 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff00413ff40 0x7ff0041362a0 secure :-1 s=READY pgs=5458 cs=0 l=1 rev1=1 crypto rx=0x7ff0040b5a00 tx=0x7feff40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:33.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.206+0000 7feffaffd640 10 client.?.objecter ms_handle_connect 0x7ff00413ff40 2024-05-15T13:20:33.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.206+0000 7feffaffd640 10 client.?.objecter resend_mon_ops 2024-05-15T13:20:33.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.209+0000 7feffaffd640 1 -- 172.21.15.117:0/3872554962 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff4035040 con 0x7ff00413ff40 2024-05-15T13:20:33.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.210+0000 7feffaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:33.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.210+0000 7feffaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T13:20:33.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.210+0000 7feffaffd640 10 monclient: dump: 2024-05-15T13:20:33.209 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:33.209 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:33.209 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:33.210 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:33.210 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:33.210 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:33.210 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:33.210 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:33.210 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:33.210 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:33.210 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.210+0000 7feffaffd640 10 monclient: _finish_auth 0 2024-05-15T13:20:33.210 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.210+0000 7feffaffd640 10 monclient: _check_auth_tickets 2024-05-15T13:20:33.210 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.210+0000 7feffaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:20:33.210 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.210+0000 7feffaffd640 1 -- 172.21.15.117:0/3872554962 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feff403ad00 con 0x7ff00413ff40 2024-05-15T13:20:33.210 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.210+0000 7ff00c7de640 5 monclient: authenticate success, global_id 22204 2024-05-15T13:20:33.210 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.210+0000 7ff00c7de640 10 monclient: _renew_subs 2024-05-15T13:20:33.210 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.210+0000 7ff00c7de640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T13:20:33.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.210+0000 7ff00c7de640 1 -- 172.21.15.117:0/3872554962 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0040ac1d0 con 0x7ff00413ff40 2024-05-15T13:20:33.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.210+0000 7feffaffd640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:20:33.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.210+0000 7feffaffd640 1 -- 172.21.15.117:0/3872554962 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff40466a0 con 0x7ff00413ff40 2024-05-15T13:20:33.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.210+0000 7feffaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:33.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.210+0000 7feffaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T13:20:33.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.210+0000 7feffaffd640 10 monclient: dump: 2024-05-15T13:20:33.211 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:33.211 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:33.211 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:33.211 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:33.211 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:33.211 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:33.211 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:33.211 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:33.211 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:33.211 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:33.212 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.210+0000 7ff008d77640 4 set_mon_vals no callback set 2024-05-15T13:20:33.212 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.210+0000 7ff00c7de640 10 client.22204.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:20:33.212 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.210+0000 7ff00c7de640 10 monclient: _renew_subs 2024-05-15T13:20:33.212 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.210+0000 7ff00c7de640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T13:20:33.212 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.210+0000 7ff00c7de640 1 -- 172.21.15.117:0/3872554962 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff004138370 con 0x7ff00413ff40 2024-05-15T13:20:33.212 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.210+0000 7ff00c7de640 1 librados: init done 2024-05-15T13:20:33.212 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.210+0000 7ff00c7de640 10 client.22204.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:20:33.214 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.210+0000 7ff00c7de640 15 client.22204.objecter _session_command_op_assign -1 1 2024-05-15T13:20:33.214 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.210+0000 7ff00c7de640 20 client.22204.objecter _get_session osd=-1 returning homeless 2024-05-15T13:20:33.214 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.210+0000 7ff00c7de640 10 client.22204.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:20:33.214 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.210+0000 7ff00c7de640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T13:20:33.214 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.210+0000 7ff00c7de640 1 -- 172.21.15.117:0/3872554962 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff00404f0e0 con 0x7ff00413ff40 2024-05-15T13:20:33.227 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.228+0000 7feffaffd640 1 -- 172.21.15.117:0/3872554962 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7feff4038020 con 0x7ff00413ff40 2024-05-15T13:20:33.227 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.228+0000 7feffaffd640 1 --2- 172.21.15.117:0/3872554962 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fefe806ceb0 0x7fefe806f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:33.227 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.229+0000 7ff00a57a640 1 --2- 172.21.15.117:0/3872554962 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fefe806ceb0 0x7fefe806f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:33.228 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.229+0000 7ff00a57a640 10 monclient: get_auth_request con 0x7fefe806ceb0 auth_method 0 2024-05-15T13:20:33.228 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.229+0000 7feffaffd640 10 client.22204.objecter ms_handle_connect 0x7fefe806ceb0 2024-05-15T13:20:33.228 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.229+0000 7ff00a57a640 1 --2- 172.21.15.117:0/3872554962 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fefe806ceb0 0x7fefe806f370 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7ff000007fd0 tx=0x7ff000007d80 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:33.230 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.231+0000 7feffaffd640 1 -- 172.21.15.117:0/3872554962 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(1225..1225 src has 912..1225) v4 ==== 6365+0+0 (secure 0 0 0) 0x7feff4078e80 con 0x7ff00413ff40 2024-05-15T13:20:33.230 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.231+0000 7feffaffd640 10 client.22204.objecter ms_dispatch 0x7ff004000b90 osd_map(1225..1225 src has 912..1225) v4 2024-05-15T13:20:33.230 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.231+0000 7feffaffd640 3 client.22204.objecter handle_osd_map got epochs [1225,1225] > 0 2024-05-15T13:20:33.230 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.231+0000 7feffaffd640 3 client.22204.objecter handle_osd_map decoding full epoch 1225 2024-05-15T13:20:33.230 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.231+0000 7feffaffd640 10 client.22204.objecter checking command 1 2024-05-15T13:20:33.230 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.231+0000 7feffaffd640 1 --2- 172.21.15.117:0/3872554962 >> [v2:172.21.15.178:6800/1383036222,v1:172.21.15.178:6801/1383036222] conn(0x7fefe8074bc0 0x7fefe8077060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:33.231 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.231+0000 7feffaffd640 20 client.22204.objecter _get_session s=0x7fefe80744f0 osd=6 3 2024-05-15T13:20:33.231 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.231+0000 7feffaffd640 20 client.22204.objecter put_session s=0x7fefe80744f0 osd=6 3 2024-05-15T13:20:33.231 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.231+0000 7feffaffd640 15 client.22204.objecter _session_command_op_remove -1 1 2024-05-15T13:20:33.231 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.231+0000 7feffaffd640 20 client.22204.objecter _get_session s=0x7fefe80744f0 osd=6 3 2024-05-15T13:20:33.231 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.231+0000 7feffaffd640 20 client.22204.objecter get_session s=0x7fefe80744f0 osd=6 3 2024-05-15T13:20:33.231 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.231+0000 7feffaffd640 15 client.22204.objecter _session_command_op_assign 6 1 2024-05-15T13:20:33.231 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.231+0000 7feffaffd640 20 client.22204.objecter put_session s=0x7fefe80744f0 osd=6 4 2024-05-15T13:20:33.231 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.231+0000 7feffaffd640 10 client.22204.objecter _send_command 1 2024-05-15T13:20:33.231 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.231+0000 7feffaffd640 1 -- 172.21.15.117:0/3872554962 --> [v2:172.21.15.178:6800/1383036222,v1:172.21.15.178:6801/1383036222] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fefe8077730 con 0x7fefe8074bc0 2024-05-15T13:20:33.231 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.231+0000 7feffaffd640 20 client.22204.objecter dump_active .. 0 homeless 2024-05-15T13:20:33.232 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.232+0000 7ff00ad7b640 1 --2- 172.21.15.117:0/3872554962 >> [v2:172.21.15.178:6800/1383036222,v1:172.21.15.178:6801/1383036222] conn(0x7fefe8074bc0 0x7fefe8077060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:33.232 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.232+0000 7ff00ad7b640 10 monclient: get_auth_request con 0x7fefe8074bc0 auth_method 0 2024-05-15T13:20:33.234 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.235+0000 7ff00ad7b640 1 --2- 172.21.15.117:0/3872554962 >> [v2:172.21.15.178:6800/1383036222,v1:172.21.15.178:6801/1383036222] conn(0x7fefe8074bc0 0x7fefe8077060 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:33.234 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.235+0000 7feffaffd640 10 client.22204.objecter ms_handle_connect 0x7fefe8074bc0 2024-05-15T13:20:33.257 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.258+0000 7feffaffd640 1 -- 172.21.15.117:0/3872554962 <== osd.6 v2:172.21.15.178:6800/1383036222 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fefe8077730 con 0x7fefe8074bc0 2024-05-15T13:20:33.257 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.258+0000 7feffaffd640 10 client.22204.objecter ms_dispatch 0x7ff004000b90 command_reply(tid 1: 0 ) v1 2024-05-15T13:20:33.257 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.258+0000 7feffaffd640 10 client.22204.objecter _finish_command 1 = system:0 2024-05-15T13:20:33.257 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.259+0000 7feffaffd640 20 client.22204.objecter put_session s=0x7fefe80744f0 osd=6 4 2024-05-15T13:20:33.257 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.259+0000 7feffaffd640 15 client.22204.objecter _session_command_op_remove 6 1 2024-05-15T13:20:33.270 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.272+0000 7ff00c7de640 10 client.22204.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-05-15T13:20:33.270 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.272+0000 7ff00c7de640 15 client.22204.objecter _session_command_op_assign -1 2 2024-05-15T13:20:33.271 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.272+0000 7ff00c7de640 20 client.22204.objecter _get_session s=0x7fefe80744f0 osd=6 3 2024-05-15T13:20:33.271 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.272+0000 7ff00c7de640 20 client.22204.objecter put_session s=0x7fefe80744f0 osd=6 3 2024-05-15T13:20:33.271 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.272+0000 7ff00c7de640 20 client.22204.objecter _get_session s=0x7fefe80744f0 osd=6 3 2024-05-15T13:20:33.271 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.272+0000 7ff00c7de640 15 client.22204.objecter _session_command_op_remove -1 2 2024-05-15T13:20:33.271 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.272+0000 7ff00c7de640 20 client.22204.objecter get_session s=0x7fefe80744f0 osd=6 3 2024-05-15T13:20:33.271 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.272+0000 7ff00c7de640 15 client.22204.objecter _session_command_op_assign 6 2 2024-05-15T13:20:33.271 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.272+0000 7ff00c7de640 20 client.22204.objecter put_session s=0x7fefe80744f0 osd=6 4 2024-05-15T13:20:33.271 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.272+0000 7ff00c7de640 10 client.22204.objecter _send_command 2 2024-05-15T13:20:33.271 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.272+0000 7ff00c7de640 1 -- 172.21.15.117:0/3872554962 --> [v2:172.21.15.178:6800/1383036222,v1:172.21.15.178:6801/1383036222] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7ff004138660 con 0x7fefe8074bc0 2024-05-15T13:20:33.291 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.292+0000 7feffaffd640 1 -- 172.21.15.117:0/3872554962 <== osd.6 v2:172.21.15.178:6800/1383036222 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7ff004138660 con 0x7fefe8074bc0 2024-05-15T13:20:33.291 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.292+0000 7feffaffd640 10 client.22204.objecter ms_dispatch 0x7ff004000b90 command_reply(tid 2: 0 ) v1 2024-05-15T13:20:33.291 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.292+0000 7feffaffd640 10 client.22204.objecter _finish_command 2 = system:0 2024-05-15T13:20:33.291 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.292+0000 7feffaffd640 20 client.22204.objecter put_session s=0x7fefe80744f0 osd=6 4 2024-05-15T13:20:33.291 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.292+0000 7feffaffd640 15 client.22204.objecter _session_command_op_remove 6 2 2024-05-15T13:20:33.291 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-05-15T13:20:33.291 INFO:teuthology.orchestra.run.smithi117.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-15T13:20:33.292 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-05-15T13:20:33.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.292+0000 7ff00c7de640 10 librados: watch_flush enter 2024-05-15T13:20:33.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.292+0000 7ff00c7de640 10 librados: watch_flush exit 2024-05-15T13:20:33.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.292+0000 7ff00c7de640 10 client.22204.objecter close_session for osd.6 2024-05-15T13:20:33.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.292+0000 7ff00c7de640 1 -- 172.21.15.117:0/3872554962 >> [v2:172.21.15.178:6800/1383036222,v1:172.21.15.178:6801/1383036222] conn(0x7fefe8074bc0 msgr2=0x7fefe8077060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:33.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.293+0000 7ff00c7de640 1 --2- 172.21.15.117:0/3872554962 >> [v2:172.21.15.178:6800/1383036222,v1:172.21.15.178:6801/1383036222] conn(0x7fefe8074bc0 0x7fefe8077060 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:33.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.293+0000 7ff00c7de640 20 client.22204.objecter put_session s=0x7fefe80744f0 osd=6 1 2024-05-15T13:20:33.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.293+0000 7ff00c7de640 20 client.22204.objecter shutdown clearing up homeless session... 2024-05-15T13:20:33.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.293+0000 7ff00c7de640 10 client.22204.objecter successfully canceled tick 2024-05-15T13:20:33.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.293+0000 7ff00c7de640 1 -- 172.21.15.117:0/3872554962 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fefe806ceb0 msgr2=0x7fefe806f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:33.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.293+0000 7ff00c7de640 1 --2- 172.21.15.117:0/3872554962 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7fefe806ceb0 0x7fefe806f370 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7ff000007fd0 tx=0x7ff000007d80 comp rx=0 tx=0).stop 2024-05-15T13:20:33.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.293+0000 7ff00c7de640 10 monclient: shutdown 2024-05-15T13:20:33.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.293+0000 7ff00c7de640 20 monclient: shutdown canceling and discarding version request 1 2024-05-15T13:20:33.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.293+0000 7ff00c7de640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:20:33.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.293+0000 7ff00c7de640 1 -- 172.21.15.117:0/3872554962 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff00413ff40 msgr2=0x7ff0041362a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:33.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.293+0000 7ff00c7de640 1 --2- 172.21.15.117:0/3872554962 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff00413ff40 0x7ff0041362a0 secure :-1 s=READY pgs=5458 cs=0 l=1 rev1=1 crypto rx=0x7ff0040b5a00 tx=0x7feff40046c0 comp rx=0 tx=0).stop 2024-05-15T13:20:33.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.293+0000 7ff00ad7b640 1 -- 172.21.15.117:0/3872554962 reap_dead start 2024-05-15T13:20:33.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.293+0000 7ff00c7de640 1 -- 172.21.15.117:0/3872554962 shutdown_connections 2024-05-15T13:20:33.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.293+0000 7ff00c7de640 1 -- 172.21.15.117:0/3872554962 >> 172.21.15.117:0/3872554962 conn(0x7ff004066ed0 msgr2=0x7ff00413a6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:20:33.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.293+0000 7ff00c7de640 1 -- 172.21.15.117:0/3872554962 shutdown_connections 2024-05-15T13:20:33.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.293+0000 7ff00c7de640 1 -- 172.21.15.117:0/3872554962 wait complete. 2024-05-15T13:20:33.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.293+0000 7ff00c7de640 1 librados: shutdown 2024-05-15T13:20:33.303 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-05-15T13:20:33.410 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.412+0000 7f874631b640 10 monclient: get_monmap_and_config 2024-05-15T13:20:33.411 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.412+0000 7f874631b640 10 monclient: build_initial_monmap 2024-05-15T13:20:33.411 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.412+0000 7f874631b640 10 monclient: monmap: 2024-05-15T13:20:33.411 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T13:20:33.411 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:33.411 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T13:20:33.413149+0000 2024-05-15T13:20:33.411 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T13:20:33.413149+0000 2024-05-15T13:20:33.411 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T13:20:33.411 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:33.411 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T13:20:33.411 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T13:20:33.411 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T13:20:33.411 INFO:teuthology.orchestra.run.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T13:20:33.411 INFO:teuthology.orchestra.run.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T13:20:33.411 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:33.413 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.413+0000 7f874631b640 1 Processor -- start 2024-05-15T13:20:33.413 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.413+0000 7f874631b640 1 -- start start 2024-05-15T13:20:33.413 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.413+0000 7f874631b640 10 monclient: init 2024-05-15T13:20:33.413 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.414+0000 7f874631b640 10 monclient: _reopen_session rank -1 2024-05-15T13:20:33.413 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.414+0000 7f874631b640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-05-15T13:20:33.413 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.414+0000 7f874631b640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f87400b86e0 0x7f87400aa0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:33.413 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.414+0000 7f874631b640 10 monclient(hunting): picked mon.noname-a con 0x7f87400b86e0 addr v2:172.21.15.117:3300/0 2024-05-15T13:20:33.413 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.414+0000 7f874631b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f87400b1dd0 addr v1:172.21.15.178:6789/0 2024-05-15T13:20:33.413 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.414+0000 7f874631b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f874013f870 addr v1:172.21.15.117:6789/0 2024-05-15T13:20:33.413 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.414+0000 7f874631b640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87400a9bc0 con 0x7f874013f870 2024-05-15T13:20:33.414 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.414+0000 7f874631b640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87400a9850 con 0x7f87400b1dd0 2024-05-15T13:20:33.414 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.414+0000 7f874631b640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:33.414 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.414+0000 7f874631b640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87400b8ac0 con 0x7f87400b86e0 2024-05-15T13:20:33.414 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.414+0000 7f874631b640 10 monclient(hunting): _renew_subs 2024-05-15T13:20:33.414 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.414+0000 7f874631b640 10 monclient(hunting): authenticate will time out at 3596.656048s 2024-05-15T13:20:33.421 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.423+0000 7f873ffff640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f87400b86e0 0x7f87400aa0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:33.422 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.423+0000 7f873ffff640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f87400b86e0 0x7f87400aa0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:44334/0 (socket says 172.21.15.117:44334) 2024-05-15T13:20:33.422 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.423+0000 7f873ffff640 1 -- 172.21.15.117:0/2438478004 learned_addr learned my addr 172.21.15.117:0/2438478004 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T13:20:33.422 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.423+0000 7f873ffff640 10 monclient(hunting): get_auth_request con 0x7f87400b86e0 auth_method 0 2024-05-15T13:20:33.422 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.423+0000 7f873ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:33.422 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.423+0000 7f873ffff640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:33.422 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.423+0000 7f873ffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:33.423 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.424+0000 7f873ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:33.423 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.424+0000 7f873ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:33.423 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.424+0000 7f873ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:33.427 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.429+0000 7f873ffff640 10 monclient(hunting): handle_auth_done global_id 16089 payload 274 2024-05-15T13:20:33.427 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.429+0000 7f873ffff640 1 -- 172.21.15.117:0/2438478004 >> v1:172.21.15.178:6789/0 conn(0x7f87400b1dd0 legacy=0x7f8740145bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:33.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.429+0000 7f873ffff640 1 -- 172.21.15.117:0/2438478004 >> v1:172.21.15.117:6789/0 conn(0x7f874013f870 legacy=0x7f874013fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T13:20:33.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.429+0000 7f873ffff640 10 monclient: _finish_hunting 0 2024-05-15T13:20:33.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.429+0000 7f873ffff640 1 monclient: found mon.noname-a 2024-05-15T13:20:33.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.429+0000 7f873ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:20:33.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.429+0000 7f873ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.117:3300/0 2024-05-15T13:20:33.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.429+0000 7f873ffff640 1 -- 172.21.15.117:0/2438478004 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87400b3a60 con 0x7f87400b86e0 2024-05-15T13:20:33.431 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.433+0000 7f873ffff640 1 --2- 172.21.15.117:0/2438478004 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f87400b86e0 0x7f87400aa0e0 secure :-1 s=READY pgs=3307 cs=0 l=1 rev1=1 crypto rx=0x7f8730009870 tx=0x7f873002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=671338b32193aeb7 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:33.469 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.470+0000 7f873f7fe640 1 -- 172.21.15.117:0/2438478004 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8730035020 con 0x7f87400b86e0 2024-05-15T13:20:33.469 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.470+0000 7f873f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:33.470 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.470+0000 7f873f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T13:20:33.470 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.470+0000 7f873f7fe640 10 monclient: dump: 2024-05-15T13:20:33.470 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:33.470 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:33.470 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:33.470 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:33.470 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:33.470 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:33.470 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:33.470 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:33.470 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:33.470 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:33.470 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.470+0000 7f873f7fe640 10 monclient: _finish_auth 0 2024-05-15T13:20:33.470 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.470+0000 7f873f7fe640 10 monclient: _check_auth_tickets 2024-05-15T13:20:33.471 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.470+0000 7f873f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:20:33.471 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.470+0000 7f873f7fe640 1 -- 172.21.15.117:0/2438478004 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8730031870 con 0x7f87400b86e0 2024-05-15T13:20:33.471 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.470+0000 7f874631b640 5 monclient: authenticate success, global_id 16089 2024-05-15T13:20:33.471 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.470+0000 7f874631b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T13:20:33.471 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.470+0000 7f873f7fe640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:20:33.471 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.470+0000 7f873f7fe640 1 -- 172.21.15.117:0/2438478004 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8730031c40 con 0x7f87400b86e0 2024-05-15T13:20:33.471 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.470+0000 7f874631b640 10 monclient: get_monmap_and_config success 2024-05-15T13:20:33.471 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.470+0000 7f873f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:33.471 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.470+0000 7f874631b640 4 set_mon_vals no callback set 2024-05-15T13:20:33.471 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.470+0000 7f873f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T13:20:33.471 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.470+0000 7f873f7fe640 10 monclient: dump: 2024-05-15T13:20:33.471 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:33.471 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:33.471 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:33.471 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:33.472 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:33.472 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:33.472 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:33.472 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:33.472 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:33.472 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:33.476 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.471+0000 7f874631b640 10 monclient: shutdown 2024-05-15T13:20:33.476 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.471+0000 7f874631b640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:20:33.476 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.471+0000 7f874631b640 1 -- 172.21.15.117:0/2438478004 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f87400b86e0 msgr2=0x7f87400aa0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:33.476 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.471+0000 7f874631b640 1 --2- 172.21.15.117:0/2438478004 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f87400b86e0 0x7f87400aa0e0 secure :-1 s=READY pgs=3307 cs=0 l=1 rev1=1 crypto rx=0x7f8730009870 tx=0x7f873002b910 comp rx=0 tx=0).stop 2024-05-15T13:20:33.476 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.471+0000 7f874631b640 1 -- 172.21.15.117:0/2438478004 shutdown_connections 2024-05-15T13:20:33.476 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.471+0000 7f874631b640 1 --2- 172.21.15.117:0/2438478004 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f87400b86e0 0x7f87400aa0e0 unknown :-1 s=CLOSED pgs=3307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:33.476 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.471+0000 7f874631b640 1 -- 172.21.15.117:0/2438478004 >> 172.21.15.117:0/2438478004 conn(0x7f8740066f00 msgr2=0x7f8740067310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:20:33.476 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.471+0000 7f874631b640 1 -- 172.21.15.117:0/2438478004 shutdown_connections 2024-05-15T13:20:33.476 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.471+0000 7f874631b640 1 -- 172.21.15.117:0/2438478004 wait complete. 2024-05-15T13:20:33.476 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.472+0000 7f874631b640 10 monclient: build_initial_monmap 2024-05-15T13:20:33.476 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.472+0000 7f874631b640 10 monclient: monmap: 2024-05-15T13:20:33.476 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T13:20:33.477 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T13:20:33.477 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-05-15T13:20:33.477 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-05-15T13:20:33.477 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T13:20:33.477 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:33.477 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T13:20:33.477 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T13:20:33.477 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T13:20:33.477 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:33.477 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.472+0000 7f874631b640 1 librados: starting msgr at 2024-05-15T13:20:33.477 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.472+0000 7f874631b640 1 librados: starting objecter 2024-05-15T13:20:33.477 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.472+0000 7f874631b640 1 Processor -- start 2024-05-15T13:20:33.477 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.472+0000 7f874631b640 1 -- start start 2024-05-15T13:20:33.477 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.472+0000 7f874631b640 1 librados: setting wanted keys 2024-05-15T13:20:33.478 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.472+0000 7f874631b640 1 librados: calling monclient init 2024-05-15T13:20:33.478 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.472+0000 7f874631b640 10 monclient: init 2024-05-15T13:20:33.478 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.473+0000 7f874631b640 10 monclient: _reopen_session rank -1 2024-05-15T13:20:33.478 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.473+0000 7f874631b640 10 monclient: _add_conns ranks=[0,1,2] 2024-05-15T13:20:33.478 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.473+0000 7f874631b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f87400b1dd0 0x7f8740136e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:33.478 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.473+0000 7f874631b640 10 monclient(hunting): picked mon.noname-a con 0x7f87400b1dd0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T13:20:33.478 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.473+0000 7f874631b640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f87400b86e0 0x7f8740135340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:33.478 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.473+0000 7f874631b640 10 monclient(hunting): picked mon.noname-b con 0x7f87400b86e0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T13:20:33.478 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.473+0000 7f874631b640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f874013f870 0x7f8740132e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:33.478 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.473+0000 7f874631b640 10 monclient(hunting): picked mon.noname-c con 0x7f874013f870 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T13:20:33.478 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.473+0000 7f874631b640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:33.478 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.473+0000 7f874631b640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f874014e130 con 0x7f87400b1dd0 2024-05-15T13:20:33.478 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.473+0000 7f874631b640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:33.478 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.473+0000 7f874631b640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87400aa620 con 0x7f87400b86e0 2024-05-15T13:20:33.479 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.473+0000 7f874631b640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:33.479 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.473+0000 7f874631b640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f874014e2b0 con 0x7f874013f870 2024-05-15T13:20:33.479 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.473+0000 7f874631b640 10 monclient(hunting): _renew_subs 2024-05-15T13:20:33.479 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.473+0000 7f874631b640 10 monclient(hunting): authenticate will time out at 3596.715259s 2024-05-15T13:20:33.479 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.473+0000 7f873ffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f87400b1dd0 0x7f8740136e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:33.479 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.474+0000 7f8737fff640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f87400b86e0 0x7f8740135340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:33.479 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.474+0000 7f87448b8640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f874013f870 0x7f8740132e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:33.479 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.474+0000 7f873ffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f87400b1dd0 0x7f8740136e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:44356/0 (socket says 172.21.15.117:44356) 2024-05-15T13:20:33.479 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.474+0000 7f873ffff640 1 -- 172.21.15.117:0/478199949 learned_addr learned my addr 172.21.15.117:0/478199949 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T13:20:33.479 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.474+0000 7f873ffff640 10 monclient(hunting): get_auth_request con 0x7f87400b1dd0 auth_method 0 2024-05-15T13:20:33.479 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.474+0000 7f873ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:33.479 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.474+0000 7f873ffff640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:33.479 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.474+0000 7f873ffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:33.479 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.474+0000 7f8737fff640 10 monclient(hunting): get_auth_request con 0x7f87400b86e0 auth_method 0 2024-05-15T13:20:33.480 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.474+0000 7f8737fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:33.480 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.474+0000 7f8737fff640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:33.480 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.474+0000 7f8737fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:33.480 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.474+0000 7f87448b8640 10 monclient(hunting): get_auth_request con 0x7f874013f870 auth_method 0 2024-05-15T13:20:33.480 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.474+0000 7f87448b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:33.480 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.474+0000 7f87448b8640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:33.480 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.474+0000 7f87448b8640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:33.480 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.474+0000 7f8737fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:33.480 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.474+0000 7f8737fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:33.480 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.474+0000 7f8737fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:33.480 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.474+0000 7f873ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:33.480 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.474+0000 7f873ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:33.480 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.474+0000 7f873ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:33.480 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.475+0000 7f87448b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:33.480 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.475+0000 7f87448b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:33.481 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.475+0000 7f87448b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:33.481 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.477+0000 7f8737fff640 10 monclient(hunting): handle_auth_done global_id 22210 payload 824 2024-05-15T13:20:33.481 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.477+0000 7f8737fff640 1 -- 172.21.15.117:0/478199949 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f874013f870 msgr2=0x7f8740132e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:33.481 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.477+0000 7f8737fff640 1 --2- 172.21.15.117:0/478199949 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f874013f870 0x7f8740132e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:33.481 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.477+0000 7f8737fff640 1 -- 172.21.15.117:0/478199949 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f87400b1dd0 msgr2=0x7f8740136e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:33.481 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.477+0000 7f8737fff640 1 --2- 172.21.15.117:0/478199949 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f87400b1dd0 0x7f8740136e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:33.481 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.477+0000 7f8737fff640 10 monclient: _finish_hunting 0 2024-05-15T13:20:33.481 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.477+0000 7f8737fff640 1 monclient: found mon.noname-b 2024-05-15T13:20:33.481 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.477+0000 7f8737fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:20:33.481 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.477+0000 7f8737fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2024-05-15T13:20:33.481 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.477+0000 7f8737fff640 1 -- 172.21.15.117:0/478199949 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8740143380 con 0x7f87400b86e0 2024-05-15T13:20:33.483 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.479+0000 7f8737fff640 1 --2- 172.21.15.117:0/478199949 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f87400b86e0 0x7f8740135340 secure :-1 s=READY pgs=5460 cs=0 l=1 rev1=1 crypto rx=0x7f872800c340 tx=0x7f872800c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:33.483 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.479+0000 7f873d7fa640 10 client.?.objecter ms_handle_connect 0x7f87400b86e0 2024-05-15T13:20:33.483 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.479+0000 7f873d7fa640 10 client.?.objecter resend_mon_ops 2024-05-15T13:20:33.483 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.483+0000 7f873d7fa640 1 -- 172.21.15.117:0/478199949 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87280052d0 con 0x7f87400b86e0 2024-05-15T13:20:33.483 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.483+0000 7f873d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:33.483 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.483+0000 7f873d7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T13:20:33.483 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.483+0000 7f873d7fa640 10 monclient: dump: 2024-05-15T13:20:33.483 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:33.483 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:33.483 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:33.483 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:33.484 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:33.484 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:33.484 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:33.484 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:33.484 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:33.484 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:33.484 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.483+0000 7f873d7fa640 10 monclient: _finish_auth 0 2024-05-15T13:20:33.484 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.483+0000 7f873d7fa640 10 monclient: _check_auth_tickets 2024-05-15T13:20:33.484 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.483+0000 7f873d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:20:33.484 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.483+0000 7f873d7fa640 1 -- 172.21.15.117:0/478199949 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8728005490 con 0x7f87400b86e0 2024-05-15T13:20:33.484 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.483+0000 7f874631b640 5 monclient: authenticate success, global_id 22210 2024-05-15T13:20:33.484 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.483+0000 7f874631b640 10 monclient: _renew_subs 2024-05-15T13:20:33.484 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.483+0000 7f874631b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T13:20:33.484 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.483+0000 7f874631b640 1 -- 172.21.15.117:0/478199949 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8740136480 con 0x7f87400b86e0 2024-05-15T13:20:33.484 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.483+0000 7f873d7fa640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:20:33.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.483+0000 7f873d7fa640 1 -- 172.21.15.117:0/478199949 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8728007260 con 0x7f87400b86e0 2024-05-15T13:20:33.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.483+0000 7f873d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:33.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.483+0000 7f873d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T13:20:33.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.483+0000 7f873d7fa640 10 monclient: dump: 2024-05-15T13:20:33.485 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:33.485 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:33.485 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:33.485 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:33.485 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:33.485 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:33.485 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:33.485 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:33.485 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:33.485 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:33.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.483+0000 7f873effd640 4 set_mon_vals no callback set 2024-05-15T13:20:33.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.483+0000 7f874631b640 10 client.22210.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:20:33.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.483+0000 7f874631b640 10 monclient: _renew_subs 2024-05-15T13:20:33.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.483+0000 7f874631b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T13:20:33.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.483+0000 7f874631b640 1 -- 172.21.15.117:0/478199949 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8740136a40 con 0x7f87400b86e0 2024-05-15T13:20:33.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.483+0000 7f874631b640 1 librados: init done 2024-05-15T13:20:33.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.484+0000 7f874631b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T13:20:33.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.484+0000 7f874631b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:20:33.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.484+0000 7f874631b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T13:20:33.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.484+0000 7f874631b640 1 -- 172.21.15.117:0/478199949 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f874004f0e0 con 0x7f87400b86e0 2024-05-15T13:20:33.732 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.732+0000 7f873d7fa640 1 -- 172.21.15.117:0/478199949 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f8728007400 con 0x7f87400b86e0 2024-05-15T13:20:33.732 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.733+0000 7f873d7fa640 1 --2- 172.21.15.117:0/478199949 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f872006ce60 0x7f872006f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:33.732 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.733+0000 7f873ffff640 1 --2- 172.21.15.117:0/478199949 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f872006ce60 0x7f872006f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:33.732 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.733+0000 7f873ffff640 10 monclient: get_auth_request con 0x7f872006ce60 auth_method 0 2024-05-15T13:20:33.733 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.733+0000 7f873ffff640 1 --2- 172.21.15.117:0/478199949 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f872006ce60 0x7f872006f320 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f873002b950 tx=0x7f873003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:33.733 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.733+0000 7f873d7fa640 10 client.22210.objecter ms_handle_connect 0x7f872006ce60 2024-05-15T13:20:33.736 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.737+0000 7f873d7fa640 1 -- 172.21.15.117:0/478199949 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(1225..1225 src has 912..1225) v4 ==== 6365+0+0 (secure 0 0 0) 0x7f87280563f0 con 0x7f87400b86e0 2024-05-15T13:20:33.736 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.737+0000 7f873d7fa640 10 client.22210.objecter ms_dispatch 0x7f8740000b90 osd_map(1225..1225 src has 912..1225) v4 2024-05-15T13:20:33.736 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.737+0000 7f873d7fa640 3 client.22210.objecter handle_osd_map got epochs [1225,1225] > 0 2024-05-15T13:20:33.736 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.737+0000 7f873d7fa640 3 client.22210.objecter handle_osd_map decoding full epoch 1225 2024-05-15T13:20:33.736 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.737+0000 7f873d7fa640 20 client.22210.objecter dump_active .. 0 homeless 2024-05-15T13:20:33.747 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.748+0000 7f873d7fa640 1 -- 172.21.15.117:0/478199949 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f8728058350 con 0x7f87400b86e0 2024-05-15T13:20:33.747 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.748+0000 7f873d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:20:33.747 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.748+0000 7f873d7fa640 10 monclient: _finish_command 1 = system:0 2024-05-15T13:20:33.923 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.924+0000 7f874631b640 1 -- 172.21.15.117:0/478199949 --> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f87400008d0 con 0x7f872006ce60 2024-05-15T13:20:33.924 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.925+0000 7f873d7fa640 1 -- 172.21.15.117:0/478199949 <== mgr.4107 v2:172.21.15.117:6800/1785766523 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f87400008d0 con 0x7f872006ce60 2024-05-15T13:20:33.924 INFO:teuthology.orchestra.run.smithi117.stderr:instructed osd(s) 6 to deep-scrub 2024-05-15T13:20:33.926 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.927+0000 7f874631b640 10 librados: watch_flush enter 2024-05-15T13:20:33.926 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.927+0000 7f874631b640 10 librados: watch_flush exit 2024-05-15T13:20:33.926 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.927+0000 7f874631b640 20 client.22210.objecter shutdown clearing up homeless session... 2024-05-15T13:20:33.926 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.927+0000 7f874631b640 10 client.22210.objecter successfully canceled tick 2024-05-15T13:20:33.926 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.927+0000 7f874631b640 1 -- 172.21.15.117:0/478199949 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f872006ce60 msgr2=0x7f872006f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:33.926 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.927+0000 7f874631b640 1 --2- 172.21.15.117:0/478199949 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f872006ce60 0x7f872006f320 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f873002b950 tx=0x7f873003c040 comp rx=0 tx=0).stop 2024-05-15T13:20:33.926 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.927+0000 7f874631b640 10 monclient: shutdown 2024-05-15T13:20:33.927 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.927+0000 7f874631b640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:20:33.927 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.927+0000 7f874631b640 1 -- 172.21.15.117:0/478199949 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f87400b86e0 msgr2=0x7f8740135340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:33.927 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.927+0000 7f874631b640 1 --2- 172.21.15.117:0/478199949 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f87400b86e0 0x7f8740135340 secure :-1 s=READY pgs=5460 cs=0 l=1 rev1=1 crypto rx=0x7f872800c340 tx=0x7f872800c800 comp rx=0 tx=0).stop 2024-05-15T13:20:33.927 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.927+0000 7f874631b640 1 -- 172.21.15.117:0/478199949 shutdown_connections 2024-05-15T13:20:33.927 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.927+0000 7f874631b640 1 --2- 172.21.15.117:0/478199949 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f874013f870 0x7f8740132e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:33.927 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.927+0000 7f874631b640 1 --2- 172.21.15.117:0/478199949 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f872006ce60 0x7f872006f320 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:33.927 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.927+0000 7f874631b640 1 --2- 172.21.15.117:0/478199949 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f87400b86e0 0x7f8740135340 unknown :-1 s=CLOSED pgs=5460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:33.927 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.927+0000 7f874631b640 1 --2- 172.21.15.117:0/478199949 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f87400b1dd0 0x7f8740136e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:33.927 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.927+0000 7f874631b640 1 -- 172.21.15.117:0/478199949 >> 172.21.15.117:0/478199949 conn(0x7f8740066f00 msgr2=0x7f874013c2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:20:33.927 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.927+0000 7f874631b640 1 -- 172.21.15.117:0/478199949 shutdown_connections 2024-05-15T13:20:33.927 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.927+0000 7f874631b640 1 -- 172.21.15.117:0/478199949 wait complete. 2024-05-15T13:20:33.927 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:33.927+0000 7f874631b640 1 librados: shutdown 2024-05-15T13:20:33.937 INFO:tasks.ceph:Scrubbing osd.7 2024-05-15T13:20:33.937 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-05-15T13:20:34.045 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.046+0000 7f921c4c1640 10 monclient: get_monmap_and_config 2024-05-15T13:20:34.045 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.046+0000 7f921c4c1640 10 monclient: build_initial_monmap 2024-05-15T13:20:34.045 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.046+0000 7f921c4c1640 10 monclient: monmap: 2024-05-15T13:20:34.045 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T13:20:34.046 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:34.046 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T13:20:34.047548+0000 2024-05-15T13:20:34.046 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T13:20:34.047548+0000 2024-05-15T13:20:34.046 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T13:20:34.046 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:34.046 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T13:20:34.046 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T13:20:34.046 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T13:20:34.046 INFO:teuthology.orchestra.run.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T13:20:34.046 INFO:teuthology.orchestra.run.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T13:20:34.046 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:34.047 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.047+0000 7f921c4c1640 1 Processor -- start 2024-05-15T13:20:34.047 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.047+0000 7f921c4c1640 1 -- start start 2024-05-15T13:20:34.047 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.047+0000 7f921c4c1640 10 monclient: init 2024-05-15T13:20:34.048 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.047+0000 7f921c4c1640 10 monclient: _reopen_session rank -1 2024-05-15T13:20:34.048 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.047+0000 7f921c4c1640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-05-15T13:20:34.048 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.047+0000 7f921c4c1640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f921406b010 0x7f9214143a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:34.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.047+0000 7f921c4c1640 10 monclient(hunting): picked mon.noname-b con 0x7f921406b010 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T13:20:34.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.047+0000 7f921c4c1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f92140b30c0 addr v1:172.21.15.178:6789/0 2024-05-15T13:20:34.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.047+0000 7f921c4c1640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f9214143f80 0x7f9214071a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:34.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.047+0000 7f921c4c1640 10 monclient(hunting): picked mon.noname-c con 0x7f9214143f80 addr v2:172.21.15.178:3300/0 2024-05-15T13:20:34.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.047+0000 7f921c4c1640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92140b4740 con 0x7f92140b30c0 2024-05-15T13:20:34.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.047+0000 7f921c4c1640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:34.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.047+0000 7f921c4c1640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f921406b3f0 con 0x7f9214143f80 2024-05-15T13:20:34.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.047+0000 7f921c4c1640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:34.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.047+0000 7f921c4c1640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9214144430 con 0x7f921406b010 2024-05-15T13:20:34.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.047+0000 7f921c4c1640 10 monclient(hunting): _renew_subs 2024-05-15T13:20:34.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.047+0000 7f921c4c1640 10 monclient(hunting): authenticate will time out at 3597.289726s 2024-05-15T13:20:34.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.048+0000 7f921a25d640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f921406b010 0x7f9214143a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:34.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.048+0000 7f921aa5e640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f9214143f80 0x7f9214071a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:34.050 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.048+0000 7f921a25d640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f921406b010 0x7f9214143a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.117:50442/0 (socket says 172.21.15.117:50442) 2024-05-15T13:20:34.050 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.048+0000 7f921aa5e640 10 monclient(hunting): get_auth_request con 0x7f9214143f80 auth_method 0 2024-05-15T13:20:34.050 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.048+0000 7f921a25d640 1 -- 172.21.15.117:0/170788175 learned_addr learned my addr 172.21.15.117:0/170788175 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T13:20:34.050 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.048+0000 7f921aa5e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:34.050 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.048+0000 7f921aa5e640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:34.050 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.048+0000 7f921aa5e640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:34.050 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.048+0000 7f921a25d640 10 monclient(hunting): get_auth_request con 0x7f921406b010 auth_method 0 2024-05-15T13:20:34.050 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.048+0000 7f921a25d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:34.050 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.048+0000 7f921a25d640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:34.050 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.048+0000 7f921a25d640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:34.050 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.049+0000 7f921aa5e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:34.050 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.049+0000 7f921aa5e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:34.051 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.049+0000 7f921aa5e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:34.051 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.049+0000 7f921a25d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:34.051 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.049+0000 7f921a25d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:34.051 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.049+0000 7f921a25d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:34.052 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.050+0000 7f921a25d640 10 monclient(hunting): handle_auth_done global_id 18779 payload 274 2024-05-15T13:20:34.052 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.050+0000 7f921a25d640 1 -- 172.21.15.117:0/170788175 >> v2:172.21.15.178:3300/0 conn(0x7f9214143f80 msgr2=0x7f9214071a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T13:20:34.052 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.050+0000 7f921a25d640 1 --2- 172.21.15.117:0/170788175 >> v2:172.21.15.178:3300/0 conn(0x7f9214143f80 0x7f9214071a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:34.052 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.050+0000 7f921a25d640 1 -- 172.21.15.117:0/170788175 >> v1:172.21.15.178:6789/0 conn(0x7f92140b30c0 legacy=0x7f9214071320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T13:20:34.052 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.050+0000 7f921a25d640 10 monclient: _finish_hunting 0 2024-05-15T13:20:34.053 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.050+0000 7f921a25d640 1 monclient: found mon.noname-b 2024-05-15T13:20:34.053 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.050+0000 7f921a25d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:20:34.053 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.050+0000 7f921a25d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.117:3301/0 2024-05-15T13:20:34.053 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.050+0000 7f921a25d640 1 -- 172.21.15.117:0/170788175 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9214072f00 con 0x7f921406b010 2024-05-15T13:20:34.053 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.053+0000 7f921a25d640 1 --2- 172.21.15.117:0/170788175 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f921406b010 0x7f9214143a40 secure :-1 s=READY pgs=3388 cs=0 l=1 rev1=1 crypto rx=0x7f9210009870 tx=0x7f921002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e598fdf828330f55 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:34.056 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.056+0000 7f921925b640 1 -- 172.21.15.117:0/170788175 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9210035020 con 0x7f921406b010 2024-05-15T13:20:34.056 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.056+0000 7f921925b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:34.056 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.056+0000 7f921925b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T13:20:34.056 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.056+0000 7f921925b640 10 monclient: dump: 2024-05-15T13:20:34.056 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:34.056 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:34.057 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:34.057 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:34.057 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:34.057 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:34.057 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:34.057 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:34.057 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:34.058 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:34.058 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.056+0000 7f921925b640 10 monclient: _finish_auth 0 2024-05-15T13:20:34.058 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.056+0000 7f921925b640 10 monclient: _check_auth_tickets 2024-05-15T13:20:34.058 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.056+0000 7f921925b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:20:34.058 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.057+0000 7f921925b640 1 -- 172.21.15.117:0/170788175 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9210031640 con 0x7f921406b010 2024-05-15T13:20:34.058 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.057+0000 7f921925b640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:20:34.058 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.057+0000 7f921925b640 1 -- 172.21.15.117:0/170788175 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f921002d040 con 0x7f921406b010 2024-05-15T13:20:34.058 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.057+0000 7f921925b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:34.058 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.057+0000 7f921925b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T13:20:34.058 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.057+0000 7f921925b640 10 monclient: dump: 2024-05-15T13:20:34.059 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:34.059 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:34.059 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:34.059 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:34.059 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:34.059 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:34.059 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:34.059 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:34.059 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:34.059 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:34.059 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.057+0000 7f921c4c1640 5 monclient: authenticate success, global_id 18779 2024-05-15T13:20:34.059 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.057+0000 7f921c4c1640 10 monclient: get_monmap_and_config success 2024-05-15T13:20:34.059 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.057+0000 7f921c4c1640 4 set_mon_vals no callback set 2024-05-15T13:20:34.059 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.057+0000 7f921c4c1640 10 monclient: shutdown 2024-05-15T13:20:34.059 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.057+0000 7f921c4c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:20:34.060 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.057+0000 7f921c4c1640 1 -- 172.21.15.117:0/170788175 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f921406b010 msgr2=0x7f9214143a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:34.060 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.057+0000 7f921c4c1640 1 --2- 172.21.15.117:0/170788175 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f921406b010 0x7f9214143a40 secure :-1 s=READY pgs=3388 cs=0 l=1 rev1=1 crypto rx=0x7f9210009870 tx=0x7f921002b8f0 comp rx=0 tx=0).stop 2024-05-15T13:20:34.060 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.057+0000 7f921c4c1640 1 -- 172.21.15.117:0/170788175 shutdown_connections 2024-05-15T13:20:34.060 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.057+0000 7f921c4c1640 1 --2- 172.21.15.117:0/170788175 >> v2:172.21.15.178:3300/0 conn(0x7f9214143f80 0x7f9214071a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:34.060 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.057+0000 7f921c4c1640 1 --2- 172.21.15.117:0/170788175 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f921406b010 0x7f9214143a40 unknown :-1 s=CLOSED pgs=3388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:34.060 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.057+0000 7f921c4c1640 1 -- 172.21.15.117:0/170788175 >> 172.21.15.117:0/170788175 conn(0x7f92140b19b0 msgr2=0x7f92140b1dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:20:34.060 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.057+0000 7f921c4c1640 1 -- 172.21.15.117:0/170788175 shutdown_connections 2024-05-15T13:20:34.060 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.057+0000 7f921c4c1640 1 -- 172.21.15.117:0/170788175 wait complete. 2024-05-15T13:20:34.064 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.058+0000 7f921c4c1640 10 monclient: build_initial_monmap 2024-05-15T13:20:34.064 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.058+0000 7f921c4c1640 10 monclient: monmap: 2024-05-15T13:20:34.064 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T13:20:34.064 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T13:20:34.064 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-05-15T13:20:34.064 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-05-15T13:20:34.064 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T13:20:34.064 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:34.064 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T13:20:34.064 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T13:20:34.065 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T13:20:34.065 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:34.065 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.058+0000 7f921c4c1640 1 librados: starting msgr at 2024-05-15T13:20:34.065 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.058+0000 7f921c4c1640 1 librados: starting objecter 2024-05-15T13:20:34.065 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.058+0000 7f921c4c1640 1 Processor -- start 2024-05-15T13:20:34.065 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.058+0000 7f921c4c1640 1 -- start start 2024-05-15T13:20:34.065 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.058+0000 7f921c4c1640 1 librados: setting wanted keys 2024-05-15T13:20:34.065 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.058+0000 7f921c4c1640 1 librados: calling monclient init 2024-05-15T13:20:34.065 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.058+0000 7f921c4c1640 10 monclient: init 2024-05-15T13:20:34.065 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.059+0000 7f921c4c1640 10 monclient: _reopen_session rank -1 2024-05-15T13:20:34.065 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.059+0000 7f921c4c1640 10 monclient: _add_conns ranks=[1,0,2] 2024-05-15T13:20:34.065 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.059+0000 7f921c4c1640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f921406b010 0x7f92141ce2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:34.066 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.059+0000 7f921c4c1640 10 monclient(hunting): picked mon.noname-b con 0x7f921406b010 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T13:20:34.066 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.059+0000 7f921c4c1640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f92140b30c0 0x7f9214142a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:34.066 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.059+0000 7f921c4c1640 10 monclient(hunting): picked mon.noname-a con 0x7f92140b30c0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T13:20:34.066 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.059+0000 7f921c4c1640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9214143f80 0x7f92141db560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:34.066 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.059+0000 7f921c4c1640 10 monclient(hunting): picked mon.noname-c con 0x7f9214143f80 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T13:20:34.066 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.059+0000 7f921c4c1640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:34.066 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.059+0000 7f921c4c1640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92140b3c20 con 0x7f92140b30c0 2024-05-15T13:20:34.066 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.059+0000 7f921c4c1640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:34.066 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.059+0000 7f921c4c1640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92140b3aa0 con 0x7f921406b010 2024-05-15T13:20:34.066 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.059+0000 7f921c4c1640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:34.066 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.059+0000 7f921c4c1640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9214071f70 con 0x7f9214143f80 2024-05-15T13:20:34.066 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.059+0000 7f921c4c1640 10 monclient(hunting): _renew_subs 2024-05-15T13:20:34.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.059+0000 7f921c4c1640 10 monclient(hunting): authenticate will time out at 3597.301259s 2024-05-15T13:20:34.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.059+0000 7f9219a5c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f92140b30c0 0x7f9214142a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:34.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.059+0000 7f921a25d640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f921406b010 0x7f92141ce2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:34.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.060+0000 7f921aa5e640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9214143f80 0x7f92141db560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:34.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.060+0000 7f921a25d640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f921406b010 0x7f92141ce2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:37474/0 (socket says 172.21.15.117:37474) 2024-05-15T13:20:34.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.060+0000 7f921a25d640 1 -- 172.21.15.117:0/3531856553 learned_addr learned my addr 172.21.15.117:0/3531856553 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T13:20:34.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.060+0000 7f921a25d640 10 monclient(hunting): get_auth_request con 0x7f921406b010 auth_method 0 2024-05-15T13:20:34.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.060+0000 7f921a25d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:34.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.060+0000 7f921a25d640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:34.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.060+0000 7f921a25d640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:34.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.060+0000 7f9219a5c640 10 monclient(hunting): get_auth_request con 0x7f92140b30c0 auth_method 0 2024-05-15T13:20:34.068 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.060+0000 7f9219a5c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:34.068 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.060+0000 7f9219a5c640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:34.068 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.060+0000 7f9219a5c640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:34.068 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.060+0000 7f921aa5e640 10 monclient(hunting): get_auth_request con 0x7f9214143f80 auth_method 0 2024-05-15T13:20:34.068 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.060+0000 7f921aa5e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:34.068 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.060+0000 7f921aa5e640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:34.068 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.060+0000 7f921aa5e640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:34.068 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.060+0000 7f921a25d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:34.069 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.060+0000 7f921a25d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:34.069 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.060+0000 7f921a25d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:34.069 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.061+0000 7f9219a5c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:34.069 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.061+0000 7f9219a5c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:34.069 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.061+0000 7f9219a5c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:34.069 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.061+0000 7f921aa5e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:34.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.061+0000 7f921aa5e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:34.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.061+0000 7f921aa5e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:34.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.063+0000 7f921a25d640 10 monclient(hunting): handle_auth_done global_id 22216 payload 824 2024-05-15T13:20:34.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.063+0000 7f921a25d640 1 -- 172.21.15.117:0/3531856553 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9214143f80 msgr2=0x7f92141db560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:34.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.063+0000 7f921a25d640 1 --2- 172.21.15.117:0/3531856553 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9214143f80 0x7f92141db560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:34.071 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.063+0000 7f921a25d640 1 -- 172.21.15.117:0/3531856553 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f92140b30c0 msgr2=0x7f9214142a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:34.071 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.063+0000 7f921a25d640 1 --2- 172.21.15.117:0/3531856553 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f92140b30c0 0x7f9214142a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:34.071 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.063+0000 7f921a25d640 10 monclient: _finish_hunting 0 2024-05-15T13:20:34.071 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.063+0000 7f921a25d640 1 monclient: found mon.noname-b 2024-05-15T13:20:34.071 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.063+0000 7f921a25d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:20:34.071 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.063+0000 7f921a25d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2024-05-15T13:20:34.072 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.063+0000 7f921a25d640 1 -- 172.21.15.117:0/3531856553 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92141424e0 con 0x7f921406b010 2024-05-15T13:20:34.072 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.065+0000 7f921a25d640 1 --2- 172.21.15.117:0/3531856553 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f921406b010 0x7f92141ce2d0 secure :-1 s=READY pgs=5461 cs=0 l=1 rev1=1 crypto rx=0x7f9210009870 tx=0x7f92100046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:34.072 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.065+0000 7f920affd640 10 client.?.objecter ms_handle_connect 0x7f921406b010 2024-05-15T13:20:34.072 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.065+0000 7f920affd640 10 client.?.objecter resend_mon_ops 2024-05-15T13:20:34.074 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.069+0000 7f920affd640 1 -- 172.21.15.117:0/3531856553 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9210040070 con 0x7f921406b010 2024-05-15T13:20:34.074 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.069+0000 7f920affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:34.074 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.069+0000 7f920affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T13:20:34.074 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.069+0000 7f920affd640 10 monclient: dump: 2024-05-15T13:20:34.074 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:34.074 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:34.074 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:34.074 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:34.074 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:34.074 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:34.074 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:34.075 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:34.075 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:34.075 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:34.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.069+0000 7f920affd640 10 monclient: _finish_auth 0 2024-05-15T13:20:34.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.069+0000 7f920affd640 10 monclient: _check_auth_tickets 2024-05-15T13:20:34.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.069+0000 7f920affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:20:34.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.069+0000 7f920affd640 1 -- 172.21.15.117:0/3531856553 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92100399f0 con 0x7f921406b010 2024-05-15T13:20:34.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.069+0000 7f920affd640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:20:34.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.069+0000 7f920affd640 1 -- 172.21.15.117:0/3531856553 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9210035040 con 0x7f921406b010 2024-05-15T13:20:34.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.069+0000 7f921c4c1640 5 monclient: authenticate success, global_id 22216 2024-05-15T13:20:34.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.069+0000 7f921c4c1640 10 monclient: _renew_subs 2024-05-15T13:20:34.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.069+0000 7f921c4c1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T13:20:34.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.069+0000 7f921c4c1640 1 -- 172.21.15.117:0/3531856553 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92141dbaa0 con 0x7f921406b010 2024-05-15T13:20:34.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.069+0000 7f920affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:34.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.069+0000 7f920affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T13:20:34.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.069+0000 7f920affd640 10 monclient: dump: 2024-05-15T13:20:34.076 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:34.076 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:34.076 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:34.076 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:34.076 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:34.076 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:34.076 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:34.076 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:34.076 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:34.076 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:34.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.069+0000 7f9218a5a640 4 set_mon_vals no callback set 2024-05-15T13:20:34.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.069+0000 7f921c4c1640 10 client.22216.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:20:34.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.069+0000 7f921c4c1640 10 monclient: _renew_subs 2024-05-15T13:20:34.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.069+0000 7f921c4c1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T13:20:34.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.069+0000 7f921c4c1640 1 -- 172.21.15.117:0/3531856553 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9214141a60 con 0x7f921406b010 2024-05-15T13:20:34.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.069+0000 7f921c4c1640 1 librados: init done 2024-05-15T13:20:34.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.069+0000 7f921c4c1640 10 client.22216.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:20:34.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.069+0000 7f921c4c1640 15 client.22216.objecter _session_command_op_assign -1 1 2024-05-15T13:20:34.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.069+0000 7f921c4c1640 20 client.22216.objecter _get_session osd=-1 returning homeless 2024-05-15T13:20:34.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.069+0000 7f921c4c1640 10 client.22216.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:20:34.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.069+0000 7f921c4c1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T13:20:34.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.069+0000 7f921c4c1640 1 -- 172.21.15.117:0/3531856553 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9214143280 con 0x7f921406b010 2024-05-15T13:20:34.086 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.087+0000 7f920affd640 1 -- 172.21.15.117:0/3531856553 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f921004b060 con 0x7f921406b010 2024-05-15T13:20:34.087 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.088+0000 7f920affd640 1 --2- 172.21.15.117:0/3531856553 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f91f806ce60 0x7f91f806f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:34.087 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.088+0000 7f9219a5c640 1 --2- 172.21.15.117:0/3531856553 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f91f806ce60 0x7f91f806f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:34.087 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.088+0000 7f9219a5c640 10 monclient: get_auth_request con 0x7f91f806ce60 auth_method 0 2024-05-15T13:20:34.087 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.088+0000 7f9219a5c640 1 --2- 172.21.15.117:0/3531856553 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f91f806ce60 0x7f91f806f320 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f92141ca5a0 tx=0x7f9204009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:34.087 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.088+0000 7f920affd640 10 client.22216.objecter ms_handle_connect 0x7f91f806ce60 2024-05-15T13:20:34.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.090+0000 7f920affd640 1 -- 172.21.15.117:0/3531856553 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(1225..1225 src has 912..1225) v4 ==== 6365+0+0 (secure 0 0 0) 0x7f9210078840 con 0x7f921406b010 2024-05-15T13:20:34.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.090+0000 7f920affd640 10 client.22216.objecter ms_dispatch 0x7f9214000b90 osd_map(1225..1225 src has 912..1225) v4 2024-05-15T13:20:34.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.090+0000 7f920affd640 3 client.22216.objecter handle_osd_map got epochs [1225,1225] > 0 2024-05-15T13:20:34.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.090+0000 7f920affd640 3 client.22216.objecter handle_osd_map decoding full epoch 1225 2024-05-15T13:20:34.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.091+0000 7f920affd640 10 client.22216.objecter checking command 1 2024-05-15T13:20:34.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.091+0000 7f920affd640 1 --2- 172.21.15.117:0/3531856553 >> [v2:172.21.15.178:6808/3049776526,v1:172.21.15.178:6809/3049776526] conn(0x7f91f8074bd0 0x7f91f8077070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:34.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.091+0000 7f920affd640 20 client.22216.objecter _get_session s=0x7f91f8074500 osd=7 3 2024-05-15T13:20:34.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.091+0000 7f920affd640 20 client.22216.objecter put_session s=0x7f91f8074500 osd=7 3 2024-05-15T13:20:34.091 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.091+0000 7f920affd640 15 client.22216.objecter _session_command_op_remove -1 1 2024-05-15T13:20:34.091 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.091+0000 7f920affd640 20 client.22216.objecter _get_session s=0x7f91f8074500 osd=7 3 2024-05-15T13:20:34.091 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.091+0000 7f920affd640 20 client.22216.objecter get_session s=0x7f91f8074500 osd=7 3 2024-05-15T13:20:34.091 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.091+0000 7f920affd640 15 client.22216.objecter _session_command_op_assign 7 1 2024-05-15T13:20:34.091 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.091+0000 7f920affd640 20 client.22216.objecter put_session s=0x7f91f8074500 osd=7 4 2024-05-15T13:20:34.091 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.091+0000 7f920affd640 10 client.22216.objecter _send_command 1 2024-05-15T13:20:34.091 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.091+0000 7f920affd640 1 -- 172.21.15.117:0/3531856553 --> [v2:172.21.15.178:6808/3049776526,v1:172.21.15.178:6809/3049776526] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f921003ce50 con 0x7f91f8074bd0 2024-05-15T13:20:34.091 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.091+0000 7f920affd640 20 client.22216.objecter dump_active .. 0 homeless 2024-05-15T13:20:34.091 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.091+0000 7f920affd640 1 -- 172.21.15.117:0/3531856553 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_get_version_reply(handle=1 version=1225) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9210076a40 con 0x7f921406b010 2024-05-15T13:20:34.091 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.091+0000 7f920affd640 10 monclient: handle_get_version_reply finishing 1 version 1225 2024-05-15T13:20:34.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.091+0000 7f921aa5e640 1 --2- 172.21.15.117:0/3531856553 >> [v2:172.21.15.178:6808/3049776526,v1:172.21.15.178:6809/3049776526] conn(0x7f91f8074bd0 0x7f91f8077070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:34.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.092+0000 7f921aa5e640 10 monclient: get_auth_request con 0x7f91f8074bd0 auth_method 0 2024-05-15T13:20:34.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.094+0000 7f920affd640 10 client.22216.objecter ms_handle_connect 0x7f91f8074bd0 2024-05-15T13:20:34.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.094+0000 7f921aa5e640 1 --2- 172.21.15.117:0/3531856553 >> [v2:172.21.15.178:6808/3049776526,v1:172.21.15.178:6809/3049776526] conn(0x7f91f8074bd0 0x7f91f8077070 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:34.116 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.118+0000 7f920affd640 1 -- 172.21.15.117:0/3531856553 <== osd.7 v2:172.21.15.178:6808/3049776526 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f921003ce50 con 0x7f91f8074bd0 2024-05-15T13:20:34.117 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.118+0000 7f920affd640 10 client.22216.objecter ms_dispatch 0x7f9214000b90 command_reply(tid 1: 0 ) v1 2024-05-15T13:20:34.117 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.118+0000 7f920affd640 10 client.22216.objecter _finish_command 1 = system:0 2024-05-15T13:20:34.117 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.118+0000 7f920affd640 20 client.22216.objecter put_session s=0x7f91f8074500 osd=7 4 2024-05-15T13:20:34.117 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.118+0000 7f920affd640 15 client.22216.objecter _session_command_op_remove 7 1 2024-05-15T13:20:34.129 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.130+0000 7f921c4c1640 10 client.22216.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-05-15T13:20:34.129 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.130+0000 7f921c4c1640 15 client.22216.objecter _session_command_op_assign -1 2 2024-05-15T13:20:34.129 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.130+0000 7f921c4c1640 20 client.22216.objecter _get_session s=0x7f91f8074500 osd=7 3 2024-05-15T13:20:34.129 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.130+0000 7f921c4c1640 20 client.22216.objecter put_session s=0x7f91f8074500 osd=7 3 2024-05-15T13:20:34.129 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.130+0000 7f921c4c1640 20 client.22216.objecter _get_session s=0x7f91f8074500 osd=7 3 2024-05-15T13:20:34.129 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.130+0000 7f921c4c1640 15 client.22216.objecter _session_command_op_remove -1 2 2024-05-15T13:20:34.129 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.130+0000 7f921c4c1640 20 client.22216.objecter get_session s=0x7f91f8074500 osd=7 3 2024-05-15T13:20:34.129 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.130+0000 7f921c4c1640 15 client.22216.objecter _session_command_op_assign 7 2 2024-05-15T13:20:34.129 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.130+0000 7f921c4c1640 20 client.22216.objecter put_session s=0x7f91f8074500 osd=7 4 2024-05-15T13:20:34.130 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.130+0000 7f921c4c1640 10 client.22216.objecter _send_command 2 2024-05-15T13:20:34.130 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.130+0000 7f921c4c1640 1 -- 172.21.15.117:0/3531856553 --> [v2:172.21.15.178:6808/3049776526,v1:172.21.15.178:6809/3049776526] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f921404f0e0 con 0x7f91f8074bd0 2024-05-15T13:20:34.143 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.145+0000 7f920affd640 1 -- 172.21.15.117:0/3531856553 <== osd.7 v2:172.21.15.178:6808/3049776526 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f921404f0e0 con 0x7f91f8074bd0 2024-05-15T13:20:34.143 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.145+0000 7f920affd640 10 client.22216.objecter ms_dispatch 0x7f9214000b90 command_reply(tid 2: 0 ) v1 2024-05-15T13:20:34.143 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.145+0000 7f920affd640 10 client.22216.objecter _finish_command 2 = system:0 2024-05-15T13:20:34.143 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.145+0000 7f920affd640 20 client.22216.objecter put_session s=0x7f91f8074500 osd=7 4 2024-05-15T13:20:34.143 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.145+0000 7f920affd640 15 client.22216.objecter _session_command_op_remove 7 2 2024-05-15T13:20:34.143 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-05-15T13:20:34.144 INFO:teuthology.orchestra.run.smithi117.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-15T13:20:34.144 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-05-15T13:20:34.145 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.145+0000 7f921c4c1640 10 librados: watch_flush enter 2024-05-15T13:20:34.145 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.145+0000 7f921c4c1640 10 librados: watch_flush exit 2024-05-15T13:20:34.145 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.145+0000 7f921c4c1640 10 client.22216.objecter close_session for osd.7 2024-05-15T13:20:34.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.145+0000 7f921c4c1640 1 -- 172.21.15.117:0/3531856553 >> [v2:172.21.15.178:6808/3049776526,v1:172.21.15.178:6809/3049776526] conn(0x7f91f8074bd0 msgr2=0x7f91f8077070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:34.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.145+0000 7f921c4c1640 1 --2- 172.21.15.117:0/3531856553 >> [v2:172.21.15.178:6808/3049776526,v1:172.21.15.178:6809/3049776526] conn(0x7f91f8074bd0 0x7f91f8077070 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:34.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.145+0000 7f921c4c1640 20 client.22216.objecter put_session s=0x7f91f8074500 osd=7 1 2024-05-15T13:20:34.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.145+0000 7f921c4c1640 20 client.22216.objecter shutdown clearing up homeless session... 2024-05-15T13:20:34.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.145+0000 7f921c4c1640 10 client.22216.objecter successfully canceled tick 2024-05-15T13:20:34.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.145+0000 7f921c4c1640 1 -- 172.21.15.117:0/3531856553 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f91f806ce60 msgr2=0x7f91f806f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:34.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.145+0000 7f921c4c1640 1 --2- 172.21.15.117:0/3531856553 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f91f806ce60 0x7f91f806f320 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f92141ca5a0 tx=0x7f9204009040 comp rx=0 tx=0).stop 2024-05-15T13:20:34.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.145+0000 7f921c4c1640 10 monclient: shutdown 2024-05-15T13:20:34.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.145+0000 7f921c4c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:20:34.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.145+0000 7f921c4c1640 1 -- 172.21.15.117:0/3531856553 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f921406b010 msgr2=0x7f92141ce2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:34.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.145+0000 7f921c4c1640 1 --2- 172.21.15.117:0/3531856553 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f921406b010 0x7f92141ce2d0 secure :-1 s=READY pgs=5461 cs=0 l=1 rev1=1 crypto rx=0x7f9210009870 tx=0x7f92100046c0 comp rx=0 tx=0).stop 2024-05-15T13:20:34.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.145+0000 7f921aa5e640 1 -- 172.21.15.117:0/3531856553 reap_dead start 2024-05-15T13:20:34.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.145+0000 7f921c4c1640 1 -- 172.21.15.117:0/3531856553 shutdown_connections 2024-05-15T13:20:34.147 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.145+0000 7f921c4c1640 1 -- 172.21.15.117:0/3531856553 >> 172.21.15.117:0/3531856553 conn(0x7f92140b19b0 msgr2=0x7f92140ade40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:20:34.147 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.145+0000 7f921c4c1640 1 -- 172.21.15.117:0/3531856553 shutdown_connections 2024-05-15T13:20:34.147 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.145+0000 7f921c4c1640 1 -- 172.21.15.117:0/3531856553 wait complete. 2024-05-15T13:20:34.147 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.145+0000 7f921c4c1640 1 librados: shutdown 2024-05-15T13:20:34.154 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-05-15T13:20:34.255 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.256+0000 7f2c6d2cf640 10 monclient: get_monmap_and_config 2024-05-15T13:20:34.256 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.256+0000 7f2c6d2cf640 10 monclient: build_initial_monmap 2024-05-15T13:20:34.256 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.256+0000 7f2c6d2cf640 10 monclient: monmap: 2024-05-15T13:20:34.256 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T13:20:34.256 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:34.256 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T13:20:34.257423+0000 2024-05-15T13:20:34.256 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T13:20:34.257423+0000 2024-05-15T13:20:34.256 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T13:20:34.256 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:34.256 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T13:20:34.256 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T13:20:34.256 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T13:20:34.256 INFO:teuthology.orchestra.run.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T13:20:34.257 INFO:teuthology.orchestra.run.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T13:20:34.257 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:34.257 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.257+0000 7f2c6d2cf640 1 Processor -- start 2024-05-15T13:20:34.257 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.257+0000 7f2c6d2cf640 1 -- start start 2024-05-15T13:20:34.257 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.257+0000 7f2c6d2cf640 10 monclient: init 2024-05-15T13:20:34.259 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.257+0000 7f2c6d2cf640 10 monclient: _reopen_session rank -1 2024-05-15T13:20:34.259 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.257+0000 7f2c6d2cf640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-05-15T13:20:34.260 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.257+0000 7f2c6d2cf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2c680b86e0 addr v1:172.21.15.117:6789/0 2024-05-15T13:20:34.260 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.257+0000 7f2c6d2cf640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f2c680b1dd0 0x7f2c680aa0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:34.260 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.257+0000 7f2c6d2cf640 10 monclient(hunting): picked mon.noname-c con 0x7f2c680b1dd0 addr v2:172.21.15.178:3300/0 2024-05-15T13:20:34.260 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.257+0000 7f2c6d2cf640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f2c6813f870 0x7f2c6813fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:34.260 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.257+0000 7f2c6d2cf640 10 monclient(hunting): picked mon.noname-a con 0x7f2c6813f870 addr v2:172.21.15.117:3300/0 2024-05-15T13:20:34.260 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.257+0000 7f2c6d2cf640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c680a9bc0 con 0x7f2c680b86e0 2024-05-15T13:20:34.260 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.257+0000 7f2c6d2cf640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:34.260 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.257+0000 7f2c6d2cf640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c680b8ac0 con 0x7f2c6813f870 2024-05-15T13:20:34.260 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.257+0000 7f2c6d2cf640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:34.260 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.257+0000 7f2c6d2cf640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c680aa620 con 0x7f2c680b1dd0 2024-05-15T13:20:34.260 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.257+0000 7f2c6d2cf640 10 monclient(hunting): _renew_subs 2024-05-15T13:20:34.260 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.257+0000 7f2c6d2cf640 10 monclient(hunting): authenticate will time out at 3597.499569s 2024-05-15T13:20:34.260 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.258+0000 7f2c677fe640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f2c6813f870 0x7f2c6813fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:34.260 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.258+0000 7f2c677fe640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f2c6813f870 0x7f2c6813fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:44376/0 (socket says 172.21.15.117:44376) 2024-05-15T13:20:34.261 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.258+0000 7f2c677fe640 1 -- 172.21.15.117:0/827992686 learned_addr learned my addr 172.21.15.117:0/827992686 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T13:20:34.261 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.258+0000 7f2c677fe640 10 monclient(hunting): get_auth_request con 0x7f2c6813f870 auth_method 0 2024-05-15T13:20:34.261 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.258+0000 7f2c677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:34.261 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.258+0000 7f2c677fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:34.261 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.258+0000 7f2c677fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:34.261 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.258+0000 7f2c5e7fc640 1 --2- 172.21.15.117:0/827992686 >> v2:172.21.15.178:3300/0 conn(0x7f2c680b1dd0 0x7f2c680aa0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:34.261 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.259+0000 7f2c677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:34.261 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.259+0000 7f2c677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:34.261 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.259+0000 7f2c677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:34.261 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.259+0000 7f2c5e7fc640 10 monclient(hunting): get_auth_request con 0x7f2c680b1dd0 auth_method 0 2024-05-15T13:20:34.261 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.259+0000 7f2c5e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:34.261 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.259+0000 7f2c5e7fc640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:34.261 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.259+0000 7f2c5e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:34.261 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.259+0000 7f2c5e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:34.261 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.259+0000 7f2c5e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:34.262 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.259+0000 7f2c5e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:34.262 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.260+0000 7f2c677fe640 10 monclient(hunting): handle_auth_done global_id 16098 payload 274 2024-05-15T13:20:34.262 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.260+0000 7f2c677fe640 1 -- 172.21.15.117:0/827992686 >> v2:172.21.15.178:3300/0 conn(0x7f2c680b1dd0 msgr2=0x7f2c680aa0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T13:20:34.262 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.260+0000 7f2c677fe640 1 --2- 172.21.15.117:0/827992686 >> v2:172.21.15.178:3300/0 conn(0x7f2c680b1dd0 0x7f2c680aa0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:34.262 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.260+0000 7f2c677fe640 1 -- 172.21.15.117:0/827992686 >> v1:172.21.15.117:6789/0 conn(0x7f2c680b86e0 legacy=0x7f2c68145bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T13:20:34.262 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.260+0000 7f2c677fe640 10 monclient: _finish_hunting 0 2024-05-15T13:20:34.262 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.260+0000 7f2c677fe640 1 monclient: found mon.noname-a 2024-05-15T13:20:34.262 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.260+0000 7f2c677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:20:34.262 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.260+0000 7f2c677fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.117:3300/0 2024-05-15T13:20:34.263 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.260+0000 7f2c677fe640 1 -- 172.21.15.117:0/827992686 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c680a9850 con 0x7f2c6813f870 2024-05-15T13:20:34.263 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.262+0000 7f2c677fe640 1 --2- 172.21.15.117:0/827992686 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c6813f870 0x7f2c6813fca0 secure :-1 s=READY pgs=3309 cs=0 l=1 rev1=1 crypto rx=0x7f2c6806cb80 tx=0x7f2c6002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50d565429081a63c server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:34.282 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.282+0000 7f2c667fc640 1 -- 172.21.15.117:0/827992686 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c6002c9f0 con 0x7f2c6813f870 2024-05-15T13:20:34.282 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.282+0000 7f2c667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:34.283 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.282+0000 7f2c667fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T13:20:34.283 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.282+0000 7f2c667fc640 10 monclient: dump: 2024-05-15T13:20:34.283 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:34.283 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:34.283 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:34.283 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:34.283 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:34.283 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:34.283 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:34.283 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:34.283 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:34.283 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:34.283 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.282+0000 7f2c667fc640 10 monclient: _finish_auth 0 2024-05-15T13:20:34.283 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.282+0000 7f2c667fc640 10 monclient: _check_auth_tickets 2024-05-15T13:20:34.283 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.282+0000 7f2c667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:20:34.284 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.282+0000 7f2c667fc640 1 -- 172.21.15.117:0/827992686 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c6002cb90 con 0x7f2c6813f870 2024-05-15T13:20:34.284 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.282+0000 7f2c667fc640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:20:34.284 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.282+0000 7f2c667fc640 1 -- 172.21.15.117:0/827992686 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c600343f0 con 0x7f2c6813f870 2024-05-15T13:20:34.284 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.282+0000 7f2c667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:34.284 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.282+0000 7f2c667fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T13:20:34.284 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.282+0000 7f2c667fc640 10 monclient: dump: 2024-05-15T13:20:34.284 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:34.284 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:34.284 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:34.285 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:34.285 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:34.285 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:34.285 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:34.285 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:34.285 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:34.285 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:34.285 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.282+0000 7f2c6d2cf640 5 monclient: authenticate success, global_id 16098 2024-05-15T13:20:34.285 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.282+0000 7f2c6d2cf640 10 monclient: get_monmap_and_config success 2024-05-15T13:20:34.285 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.282+0000 7f2c6d2cf640 4 set_mon_vals no callback set 2024-05-15T13:20:34.285 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.283+0000 7f2c6d2cf640 10 monclient: shutdown 2024-05-15T13:20:34.285 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.283+0000 7f2c6d2cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:20:34.285 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.283+0000 7f2c6d2cf640 1 -- 172.21.15.117:0/827992686 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c6813f870 msgr2=0x7f2c6813fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:34.285 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.283+0000 7f2c6d2cf640 1 --2- 172.21.15.117:0/827992686 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c6813f870 0x7f2c6813fca0 secure :-1 s=READY pgs=3309 cs=0 l=1 rev1=1 crypto rx=0x7f2c6806cb80 tx=0x7f2c6002be50 comp rx=0 tx=0).stop 2024-05-15T13:20:34.286 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.283+0000 7f2c6d2cf640 1 -- 172.21.15.117:0/827992686 shutdown_connections 2024-05-15T13:20:34.286 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.283+0000 7f2c6d2cf640 1 --2- 172.21.15.117:0/827992686 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c6813f870 0x7f2c6813fca0 unknown :-1 s=CLOSED pgs=3309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:34.286 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.283+0000 7f2c6d2cf640 1 --2- 172.21.15.117:0/827992686 >> v2:172.21.15.178:3300/0 conn(0x7f2c680b1dd0 0x7f2c680aa0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:34.286 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.283+0000 7f2c6d2cf640 1 -- 172.21.15.117:0/827992686 >> 172.21.15.117:0/827992686 conn(0x7f2c68066f00 msgr2=0x7f2c68067310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:20:34.286 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.283+0000 7f2c6d2cf640 1 -- 172.21.15.117:0/827992686 shutdown_connections 2024-05-15T13:20:34.286 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.283+0000 7f2c6d2cf640 1 -- 172.21.15.117:0/827992686 wait complete. 2024-05-15T13:20:34.286 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.284+0000 7f2c6d2cf640 10 monclient: build_initial_monmap 2024-05-15T13:20:34.286 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.284+0000 7f2c6d2cf640 10 monclient: monmap: 2024-05-15T13:20:34.286 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T13:20:34.286 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T13:20:34.286 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-05-15T13:20:34.286 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-05-15T13:20:34.286 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T13:20:34.286 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:34.287 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T13:20:34.287 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T13:20:34.287 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T13:20:34.287 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:34.290 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.284+0000 7f2c6d2cf640 1 librados: starting msgr at 2024-05-15T13:20:34.290 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.284+0000 7f2c6d2cf640 1 librados: starting objecter 2024-05-15T13:20:34.290 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.284+0000 7f2c6d2cf640 1 Processor -- start 2024-05-15T13:20:34.290 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.285+0000 7f2c6d2cf640 1 -- start start 2024-05-15T13:20:34.290 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.285+0000 7f2c6d2cf640 1 librados: setting wanted keys 2024-05-15T13:20:34.290 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.285+0000 7f2c6d2cf640 1 librados: calling monclient init 2024-05-15T13:20:34.290 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.285+0000 7f2c6d2cf640 10 monclient: init 2024-05-15T13:20:34.290 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.285+0000 7f2c6d2cf640 10 monclient: _reopen_session rank -1 2024-05-15T13:20:34.290 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.285+0000 7f2c6d2cf640 10 monclient: _add_conns ranks=[1,0,2] 2024-05-15T13:20:34.290 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.285+0000 7f2c6d2cf640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2c680b1dd0 0x7f2c68137d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:34.290 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.285+0000 7f2c6d2cf640 10 monclient(hunting): picked mon.noname-b con 0x7f2c680b1dd0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T13:20:34.291 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.285+0000 7f2c6d2cf640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c680b86e0 0x7f2c68144d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:34.291 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.285+0000 7f2c6d2cf640 10 monclient(hunting): picked mon.noname-a con 0x7f2c680b86e0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T13:20:34.291 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.285+0000 7f2c6d2cf640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2c6813f870 0x7f2c680b6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:34.291 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.285+0000 7f2c6d2cf640 10 monclient(hunting): picked mon.noname-c con 0x7f2c6813f870 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T13:20:34.291 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.285+0000 7f2c6d2cf640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:34.291 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.285+0000 7f2c6d2cf640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c680b3be0 con 0x7f2c680b86e0 2024-05-15T13:20:34.291 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.285+0000 7f2c6d2cf640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:34.291 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.285+0000 7f2c6d2cf640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c680b3a60 con 0x7f2c680b1dd0 2024-05-15T13:20:34.291 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.285+0000 7f2c6d2cf640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:34.291 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.285+0000 7f2c6d2cf640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c6814cf10 con 0x7f2c6813f870 2024-05-15T13:20:34.291 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.285+0000 7f2c6d2cf640 10 monclient(hunting): _renew_subs 2024-05-15T13:20:34.291 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.285+0000 7f2c6d2cf640 10 monclient(hunting): authenticate will time out at 3597.527387s 2024-05-15T13:20:34.291 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.286+0000 7f2c5e7fc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c680b86e0 0x7f2c68144d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:34.291 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.286+0000 7f2c66ffd640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2c680b1dd0 0x7f2c68137d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:34.291 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.286+0000 7f2c677fe640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2c6813f870 0x7f2c680b6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:34.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.286+0000 7f2c66ffd640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2c680b1dd0 0x7f2c68137d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:37498/0 (socket says 172.21.15.117:37498) 2024-05-15T13:20:34.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.286+0000 7f2c66ffd640 1 -- 172.21.15.117:0/3702493214 learned_addr learned my addr 172.21.15.117:0/3702493214 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T13:20:34.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.286+0000 7f2c66ffd640 10 monclient(hunting): get_auth_request con 0x7f2c680b1dd0 auth_method 0 2024-05-15T13:20:34.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.286+0000 7f2c66ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:34.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.286+0000 7f2c66ffd640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:34.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.286+0000 7f2c66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:34.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.286+0000 7f2c5e7fc640 10 monclient(hunting): get_auth_request con 0x7f2c680b86e0 auth_method 0 2024-05-15T13:20:34.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.286+0000 7f2c5e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:34.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.286+0000 7f2c5e7fc640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:34.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.286+0000 7f2c5e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:34.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.286+0000 7f2c677fe640 10 monclient(hunting): get_auth_request con 0x7f2c6813f870 auth_method 0 2024-05-15T13:20:34.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.286+0000 7f2c677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:34.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.286+0000 7f2c677fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:34.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.286+0000 7f2c677fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:34.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.286+0000 7f2c66ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:34.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.286+0000 7f2c66ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:34.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.286+0000 7f2c66ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:34.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.286+0000 7f2c5e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:34.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.286+0000 7f2c5e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:34.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.286+0000 7f2c5e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:34.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.287+0000 7f2c677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:34.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.287+0000 7f2c677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:34.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.287+0000 7f2c677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:34.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.289+0000 7f2c66ffd640 10 monclient(hunting): handle_auth_done global_id 22222 payload 824 2024-05-15T13:20:34.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.289+0000 7f2c66ffd640 1 -- 172.21.15.117:0/3702493214 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2c6813f870 msgr2=0x7f2c680b6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:34.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.289+0000 7f2c66ffd640 1 --2- 172.21.15.117:0/3702493214 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2c6813f870 0x7f2c680b6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:34.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.289+0000 7f2c66ffd640 1 -- 172.21.15.117:0/3702493214 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c680b86e0 msgr2=0x7f2c68144d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:34.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.289+0000 7f2c66ffd640 1 --2- 172.21.15.117:0/3702493214 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c680b86e0 0x7f2c68144d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:34.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.289+0000 7f2c66ffd640 10 monclient: _finish_hunting 0 2024-05-15T13:20:34.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.289+0000 7f2c66ffd640 1 monclient: found mon.noname-b 2024-05-15T13:20:34.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.289+0000 7f2c66ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:20:34.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.289+0000 7f2c66ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2024-05-15T13:20:34.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.289+0000 7f2c66ffd640 1 -- 172.21.15.117:0/3702493214 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c68138f40 con 0x7f2c680b1dd0 2024-05-15T13:20:34.296 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.291+0000 7f2c66ffd640 1 --2- 172.21.15.117:0/3702493214 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2c680b1dd0 0x7f2c68137d90 secure :-1 s=READY pgs=5462 cs=0 l=1 rev1=1 crypto rx=0x7f2c5800c410 tx=0x7f2c5800c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:34.296 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.291+0000 7f2c5ffff640 10 client.?.objecter ms_handle_connect 0x7f2c680b1dd0 2024-05-15T13:20:34.296 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.291+0000 7f2c5ffff640 10 client.?.objecter resend_mon_ops 2024-05-15T13:20:34.296 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.295+0000 7f2c5ffff640 1 -- 172.21.15.117:0/3702493214 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c58017070 con 0x7f2c680b1dd0 2024-05-15T13:20:34.296 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.295+0000 7f2c5ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:34.296 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.295+0000 7f2c5ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T13:20:34.296 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.295+0000 7f2c5ffff640 10 monclient: dump: 2024-05-15T13:20:34.297 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:34.297 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:34.297 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:34.297 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:34.297 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:34.297 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:34.297 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:34.298 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:34.298 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:34.298 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:34.298 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.295+0000 7f2c5ffff640 10 monclient: _finish_auth 0 2024-05-15T13:20:34.298 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.295+0000 7f2c5ffff640 10 monclient: _check_auth_tickets 2024-05-15T13:20:34.298 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.295+0000 7f2c5ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:20:34.299 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.295+0000 7f2c5ffff640 1 -- 172.21.15.117:0/3702493214 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c580035f0 con 0x7f2c680b1dd0 2024-05-15T13:20:34.299 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.295+0000 7f2c6d2cf640 5 monclient: authenticate success, global_id 22222 2024-05-15T13:20:34.299 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.296+0000 7f2c6d2cf640 10 monclient: _renew_subs 2024-05-15T13:20:34.299 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.296+0000 7f2c6d2cf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T13:20:34.299 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.296+0000 7f2c6d2cf640 1 -- 172.21.15.117:0/3702493214 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c68131e70 con 0x7f2c680b1dd0 2024-05-15T13:20:34.299 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.296+0000 7f2c5ffff640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:20:34.299 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.296+0000 7f2c5ffff640 1 -- 172.21.15.117:0/3702493214 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c58003a20 con 0x7f2c680b1dd0 2024-05-15T13:20:34.300 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.296+0000 7f2c5ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:34.300 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.296+0000 7f2c5ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T13:20:34.300 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.296+0000 7f2c5ffff640 10 monclient: dump: 2024-05-15T13:20:34.300 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:34.300 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:34.300 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:34.301 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:34.301 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:34.301 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:34.301 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:34.301 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:34.301 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:34.301 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:34.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.296+0000 7f2c657fa640 4 set_mon_vals no callback set 2024-05-15T13:20:34.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.296+0000 7f2c6d2cf640 10 client.22222.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:20:34.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.296+0000 7f2c6d2cf640 10 monclient: _renew_subs 2024-05-15T13:20:34.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.296+0000 7f2c6d2cf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T13:20:34.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.296+0000 7f2c6d2cf640 1 -- 172.21.15.117:0/3702493214 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c68145740 con 0x7f2c680b1dd0 2024-05-15T13:20:34.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.296+0000 7f2c6d2cf640 1 librados: init done 2024-05-15T13:20:34.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.297+0000 7f2c6d2cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T13:20:34.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.297+0000 7f2c6d2cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:20:34.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.297+0000 7f2c6d2cf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T13:20:34.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.297+0000 7f2c6d2cf640 1 -- 172.21.15.117:0/3702493214 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c681459a0 con 0x7f2c680b1dd0 2024-05-15T13:20:34.543 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.543+0000 7f2c5ffff640 1 -- 172.21.15.117:0/3702493214 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f2c5801a070 con 0x7f2c680b1dd0 2024-05-15T13:20:34.543 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.544+0000 7f2c5ffff640 1 --2- 172.21.15.117:0/3702493214 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f2c4806cee0 0x7f2c4806f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:34.543 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.544+0000 7f2c5e7fc640 1 --2- 172.21.15.117:0/3702493214 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f2c4806cee0 0x7f2c4806f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:34.543 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.544+0000 7f2c5e7fc640 10 monclient: get_auth_request con 0x7f2c4806cee0 auth_method 0 2024-05-15T13:20:34.543 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.544+0000 7f2c5ffff640 10 client.22222.objecter ms_handle_connect 0x7f2c4806cee0 2024-05-15T13:20:34.543 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.544+0000 7f2c5e7fc640 1 --2- 172.21.15.117:0/3702493214 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f2c4806cee0 0x7f2c4806f3a0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f2c68132380 tx=0x7f2c50009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:34.550 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.551+0000 7f2c5ffff640 1 -- 172.21.15.117:0/3702493214 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(1225..1225 src has 912..1225) v4 ==== 6365+0+0 (secure 0 0 0) 0x7f2c58056370 con 0x7f2c680b1dd0 2024-05-15T13:20:34.550 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.551+0000 7f2c5ffff640 10 client.22222.objecter ms_dispatch 0x7f2c68000b90 osd_map(1225..1225 src has 912..1225) v4 2024-05-15T13:20:34.550 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.551+0000 7f2c5ffff640 3 client.22222.objecter handle_osd_map got epochs [1225,1225] > 0 2024-05-15T13:20:34.550 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.551+0000 7f2c5ffff640 3 client.22222.objecter handle_osd_map decoding full epoch 1225 2024-05-15T13:20:34.550 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.551+0000 7f2c5ffff640 20 client.22222.objecter dump_active .. 0 homeless 2024-05-15T13:20:34.560 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.561+0000 7f2c5ffff640 1 -- 172.21.15.117:0/3702493214 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f2c58021020 con 0x7f2c680b1dd0 2024-05-15T13:20:34.560 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.561+0000 7f2c5ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:20:34.560 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.561+0000 7f2c5ffff640 10 monclient: _finish_command 1 = system:0 2024-05-15T13:20:34.764 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.765+0000 7f2c6d2cf640 1 -- 172.21.15.117:0/3702493214 --> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f2c680b22b0 con 0x7f2c4806cee0 2024-05-15T13:20:34.764 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.765+0000 7f2c5ffff640 1 -- 172.21.15.117:0/3702493214 <== mgr.4107 v2:172.21.15.117:6800/1785766523 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f2c680b22b0 con 0x7f2c4806cee0 2024-05-15T13:20:34.764 INFO:teuthology.orchestra.run.smithi117.stderr:instructed osd(s) 7 to deep-scrub 2024-05-15T13:20:34.767 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.767+0000 7f2c6d2cf640 10 librados: watch_flush enter 2024-05-15T13:20:34.767 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.767+0000 7f2c6d2cf640 10 librados: watch_flush exit 2024-05-15T13:20:34.767 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.767+0000 7f2c6d2cf640 20 client.22222.objecter shutdown clearing up homeless session... 2024-05-15T13:20:34.767 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.767+0000 7f2c6d2cf640 10 client.22222.objecter successfully canceled tick 2024-05-15T13:20:34.767 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.767+0000 7f2c6d2cf640 1 -- 172.21.15.117:0/3702493214 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f2c4806cee0 msgr2=0x7f2c4806f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:34.767 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.767+0000 7f2c6d2cf640 1 --2- 172.21.15.117:0/3702493214 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f2c4806cee0 0x7f2c4806f3a0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f2c68132380 tx=0x7f2c50009190 comp rx=0 tx=0).stop 2024-05-15T13:20:34.767 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.767+0000 7f2c6d2cf640 10 monclient: shutdown 2024-05-15T13:20:34.767 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.767+0000 7f2c6d2cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:20:34.767 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.767+0000 7f2c6d2cf640 1 -- 172.21.15.117:0/3702493214 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2c680b1dd0 msgr2=0x7f2c68137d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:34.767 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.767+0000 7f2c6d2cf640 1 --2- 172.21.15.117:0/3702493214 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2c680b1dd0 0x7f2c68137d90 secure :-1 s=READY pgs=5462 cs=0 l=1 rev1=1 crypto rx=0x7f2c5800c410 tx=0x7f2c5800c8d0 comp rx=0 tx=0).stop 2024-05-15T13:20:34.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.767+0000 7f2c6d2cf640 1 -- 172.21.15.117:0/3702493214 shutdown_connections 2024-05-15T13:20:34.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.767+0000 7f2c6d2cf640 1 --2- 172.21.15.117:0/3702493214 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2c6813f870 0x7f2c680b6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:34.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.767+0000 7f2c6d2cf640 1 --2- 172.21.15.117:0/3702493214 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c680b86e0 0x7f2c68144d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:34.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.767+0000 7f2c6d2cf640 1 --2- 172.21.15.117:0/3702493214 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f2c4806cee0 0x7f2c4806f3a0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:34.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.767+0000 7f2c6d2cf640 1 --2- 172.21.15.117:0/3702493214 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2c680b1dd0 0x7f2c68137d90 unknown :-1 s=CLOSED pgs=5462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:34.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.768+0000 7f2c6d2cf640 1 -- 172.21.15.117:0/3702493214 >> 172.21.15.117:0/3702493214 conn(0x7f2c68066f00 msgr2=0x7f2c6813b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:20:34.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.768+0000 7f2c6d2cf640 1 -- 172.21.15.117:0/3702493214 shutdown_connections 2024-05-15T13:20:34.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.768+0000 7f2c6d2cf640 1 -- 172.21.15.117:0/3702493214 wait complete. 2024-05-15T13:20:34.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.768+0000 7f2c6d2cf640 1 librados: shutdown 2024-05-15T13:20:34.777 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-15T13:20:34.877 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.878+0000 7f7c97240640 10 monclient: get_monmap_and_config 2024-05-15T13:20:34.877 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.878+0000 7f7c97240640 10 monclient: build_initial_monmap 2024-05-15T13:20:34.877 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.878+0000 7f7c97240640 10 monclient: monmap: 2024-05-15T13:20:34.877 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T13:20:34.877 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:34.877 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T13:20:34.879681+0000 2024-05-15T13:20:34.878 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T13:20:34.879681+0000 2024-05-15T13:20:34.878 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T13:20:34.878 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:34.878 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T13:20:34.878 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T13:20:34.878 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T13:20:34.878 INFO:teuthology.orchestra.run.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T13:20:34.878 INFO:teuthology.orchestra.run.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T13:20:34.878 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:34.880 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.879+0000 7f7c97240640 1 Processor -- start 2024-05-15T13:20:34.880 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.879+0000 7f7c97240640 1 -- start start 2024-05-15T13:20:34.880 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.879+0000 7f7c97240640 10 monclient: init 2024-05-15T13:20:34.880 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.879+0000 7f7c97240640 10 monclient: _reopen_session rank -1 2024-05-15T13:20:34.881 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.879+0000 7f7c97240640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-05-15T13:20:34.881 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.879+0000 7f7c97240640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f7c9006c900 0x7f7c900aeab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:34.881 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.879+0000 7f7c97240640 10 monclient(hunting): picked mon.noname-c con 0x7f7c9006c900 addr v2:172.21.15.178:3300/0 2024-05-15T13:20:34.881 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.879+0000 7f7c97240640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7c900aeff0 addr v1:172.21.15.117:6789/0 2024-05-15T13:20:34.881 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.880+0000 7f7c97240640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7c900af440 0x7f7c90146590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:34.881 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.880+0000 7f7c97240640 10 monclient(hunting): picked mon.noname-b con 0x7f7c900af440 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T13:20:34.881 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.880+0000 7f7c97240640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c900ae0d0 con 0x7f7c900aeff0 2024-05-15T13:20:34.881 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.880+0000 7f7c97240640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:34.881 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.880+0000 7f7c97240640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c9006cce0 con 0x7f7c9006c900 2024-05-15T13:20:34.881 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.880+0000 7f7c97240640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:34.881 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.880+0000 7f7c97240640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c900b0500 con 0x7f7c900af440 2024-05-15T13:20:34.881 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.880+0000 7f7c97240640 10 monclient(hunting): _renew_subs 2024-05-15T13:20:34.881 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.880+0000 7f7c97240640 10 monclient(hunting): authenticate will time out at 3598.121837s 2024-05-15T13:20:34.881 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.880+0000 7f7c8ffff640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f7c900aeff0 0x7f7c90145e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:41522/0 (socket says 172.21.15.117:41522) 2024-05-15T13:20:34.882 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.880+0000 7f7c8ffff640 1 -- 172.21.15.117:0/4171691721 learned_addr learned my addr 172.21.15.117:0/4171691721 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-05-15T13:20:34.882 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.880+0000 7f7c94fdc640 1 --2- 172.21.15.117:0/4171691721 >> v2:172.21.15.178:3300/0 conn(0x7f7c9006c900 0x7f7c900aeab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:34.882 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.880+0000 7f7c957dd640 1 --2- 172.21.15.117:0/4171691721 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7c900af440 0x7f7c90146590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:34.882 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.880+0000 7f7c94fdc640 10 monclient(hunting): get_auth_request con 0x7f7c9006c900 auth_method 0 2024-05-15T13:20:34.882 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.880+0000 7f7c94fdc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:34.882 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.880+0000 7f7c94fdc640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:34.882 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.880+0000 7f7c94fdc640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:34.882 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.881+0000 7f7c957dd640 10 monclient(hunting): get_auth_request con 0x7f7c900af440 auth_method 0 2024-05-15T13:20:34.882 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.881+0000 7f7c957dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:34.882 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.881+0000 7f7c957dd640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:34.882 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.881+0000 7f7c957dd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:34.882 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.881+0000 7f7c94fdc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:34.882 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.881+0000 7f7c94fdc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:34.883 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.881+0000 7f7c94fdc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:34.883 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.881+0000 7f7c957dd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:34.883 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.881+0000 7f7c957dd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:34.883 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.881+0000 7f7c957dd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:34.884 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.882+0000 7f7c94fdc640 10 monclient(hunting): handle_auth_done global_id 22225 payload 274 2024-05-15T13:20:34.884 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.882+0000 7f7c94fdc640 1 -- 172.21.15.117:0/4171691721 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7c900af440 msgr2=0x7f7c90146590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T13:20:34.884 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.882+0000 7f7c94fdc640 1 --2- 172.21.15.117:0/4171691721 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7c900af440 0x7f7c90146590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:34.884 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.882+0000 7f7c94fdc640 1 -- 172.21.15.117:0/4171691721 >> v1:172.21.15.117:6789/0 conn(0x7f7c900aeff0 legacy=0x7f7c90145e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:34.884 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.882+0000 7f7c94fdc640 10 monclient: _finish_hunting 0 2024-05-15T13:20:34.884 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.882+0000 7f7c94fdc640 1 monclient: found mon.noname-c 2024-05-15T13:20:34.884 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.882+0000 7f7c94fdc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:20:34.884 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.882+0000 7f7c94fdc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.178:3300/0 2024-05-15T13:20:34.884 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.882+0000 7f7c94fdc640 1 -- 172.21.15.117:0/4171691721 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c900ae3b0 con 0x7f7c9006c900 2024-05-15T13:20:34.884 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.884+0000 7f7c94fdc640 1 --2- 172.21.15.117:0/4171691721 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7c9006c900 0x7f7c900aeab0 secure :-1 s=READY pgs=5463 cs=0 l=1 rev1=1 crypto rx=0x7f7c840097a0 tx=0x7f7c8402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=637b9cad02e97f4c server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:34.887 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.888+0000 7f7c8f7fe640 1 -- 172.21.15.117:0/4171691721 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c840315a0 con 0x7f7c9006c900 2024-05-15T13:20:34.887 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.888+0000 7f7c8f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:34.887 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.888+0000 7f7c8f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T13:20:34.887 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.888+0000 7f7c8f7fe640 10 monclient: dump: 2024-05-15T13:20:34.887 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:34.887 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:34.888 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:34.888 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:34.888 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:34.888 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:34.888 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:34.888 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:34.888 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:34.888 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:34.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.888+0000 7f7c8f7fe640 10 monclient: _finish_auth 0 2024-05-15T13:20:34.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.888+0000 7f7c8f7fe640 10 monclient: _check_auth_tickets 2024-05-15T13:20:34.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.888+0000 7f7c8f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:20:34.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.888+0000 7f7c8f7fe640 1 -- 172.21.15.117:0/4171691721 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c84037070 con 0x7f7c9006c900 2024-05-15T13:20:34.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.888+0000 7f7c8f7fe640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:20:34.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.888+0000 7f7c8f7fe640 1 -- 172.21.15.117:0/4171691721 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c84031960 con 0x7f7c9006c900 2024-05-15T13:20:34.889 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.888+0000 7f7c97240640 5 monclient: authenticate success, global_id 22225 2024-05-15T13:20:34.889 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.888+0000 7f7c97240640 10 monclient: get_monmap_and_config success 2024-05-15T13:20:34.889 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.888+0000 7f7c97240640 4 set_mon_vals no callback set 2024-05-15T13:20:34.889 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.888+0000 7f7c8f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:34.889 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.888+0000 7f7c8f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T13:20:34.889 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.888+0000 7f7c8f7fe640 10 monclient: dump: 2024-05-15T13:20:34.889 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:34.889 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:34.889 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:34.889 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:34.889 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:34.889 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:34.889 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:34.889 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:34.889 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:34.889 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:34.893 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.889+0000 7f7c97240640 10 monclient: shutdown 2024-05-15T13:20:34.893 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.889+0000 7f7c97240640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:20:34.893 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.889+0000 7f7c97240640 1 -- 172.21.15.117:0/4171691721 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7c9006c900 msgr2=0x7f7c900aeab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:34.893 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.889+0000 7f7c97240640 1 --2- 172.21.15.117:0/4171691721 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7c9006c900 0x7f7c900aeab0 secure :-1 s=READY pgs=5463 cs=0 l=1 rev1=1 crypto rx=0x7f7c840097a0 tx=0x7f7c8402b840 comp rx=0 tx=0).stop 2024-05-15T13:20:34.893 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.889+0000 7f7c97240640 1 -- 172.21.15.117:0/4171691721 shutdown_connections 2024-05-15T13:20:34.893 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.889+0000 7f7c97240640 1 --2- 172.21.15.117:0/4171691721 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7c900af440 0x7f7c90146590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:34.893 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.889+0000 7f7c97240640 1 --2- 172.21.15.117:0/4171691721 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7c9006c900 0x7f7c900aeab0 unknown :-1 s=CLOSED pgs=5463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:34.893 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.889+0000 7f7c97240640 1 -- 172.21.15.117:0/4171691721 >> 172.21.15.117:0/4171691721 conn(0x7f7c900b2ac0 msgr2=0x7f7c900b2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:20:34.893 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.889+0000 7f7c97240640 1 -- 172.21.15.117:0/4171691721 shutdown_connections 2024-05-15T13:20:34.893 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.889+0000 7f7c97240640 1 -- 172.21.15.117:0/4171691721 wait complete. 2024-05-15T13:20:34.893 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.890+0000 7f7c97240640 10 monclient: build_initial_monmap 2024-05-15T13:20:34.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.890+0000 7f7c97240640 10 monclient: monmap: 2024-05-15T13:20:34.894 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T13:20:34.894 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T13:20:34.894 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-05-15T13:20:34.894 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-05-15T13:20:34.894 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T13:20:34.894 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:34.894 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T13:20:34.894 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T13:20:34.894 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T13:20:34.894 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:34.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.890+0000 7f7c97240640 1 librados: starting msgr at 2024-05-15T13:20:34.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.890+0000 7f7c97240640 1 librados: starting objecter 2024-05-15T13:20:34.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.890+0000 7f7c97240640 1 Processor -- start 2024-05-15T13:20:34.895 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.890+0000 7f7c97240640 1 -- start start 2024-05-15T13:20:34.895 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.890+0000 7f7c97240640 1 librados: setting wanted keys 2024-05-15T13:20:34.895 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.890+0000 7f7c97240640 1 librados: calling monclient init 2024-05-15T13:20:34.895 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.890+0000 7f7c97240640 10 monclient: init 2024-05-15T13:20:34.895 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.891+0000 7f7c97240640 10 monclient: _reopen_session rank -1 2024-05-15T13:20:34.895 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.891+0000 7f7c97240640 10 monclient: _add_conns ranks=[1,0,2] 2024-05-15T13:20:34.895 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.891+0000 7f7c97240640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7c9006c900 0x7f7c900b5f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:34.895 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.891+0000 7f7c97240640 10 monclient(hunting): picked mon.noname-b con 0x7f7c9006c900 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T13:20:34.895 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.891+0000 7f7c97240640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7c900aeff0 0x7f7c901cd560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:34.895 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.891+0000 7f7c97240640 10 monclient(hunting): picked mon.noname-a con 0x7f7c900aeff0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T13:20:34.895 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.891+0000 7f7c97240640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7c900af440 0x7f7c901cdaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:34.895 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.891+0000 7f7c97240640 10 monclient(hunting): picked mon.noname-c con 0x7f7c900af440 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T13:20:34.895 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.891+0000 7f7c97240640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:34.895 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.891+0000 7f7c97240640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c90146ad0 con 0x7f7c900aeff0 2024-05-15T13:20:34.895 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.891+0000 7f7c97240640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:34.896 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.891+0000 7f7c97240640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c9006d730 con 0x7f7c9006c900 2024-05-15T13:20:34.896 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.891+0000 7f7c97240640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:34.896 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.891+0000 7f7c97240640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c90146c50 con 0x7f7c900af440 2024-05-15T13:20:34.896 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.891+0000 7f7c97240640 10 monclient(hunting): _renew_subs 2024-05-15T13:20:34.896 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.891+0000 7f7c97240640 10 monclient(hunting): authenticate will time out at 3598.133223s 2024-05-15T13:20:34.896 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.891+0000 7f7c8ffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7c900aeff0 0x7f7c901cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:34.896 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.891+0000 7f7c94fdc640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7c9006c900 0x7f7c900b5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:34.896 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.892+0000 7f7c957dd640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7c900af440 0x7f7c901cdaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:34.896 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.892+0000 7f7c94fdc640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7c9006c900 0x7f7c900b5f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:37504/0 (socket says 172.21.15.117:37504) 2024-05-15T13:20:34.896 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.892+0000 7f7c94fdc640 1 -- 172.21.15.117:0/3309012988 learned_addr learned my addr 172.21.15.117:0/3309012988 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T13:20:34.896 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.892+0000 7f7c94fdc640 10 monclient(hunting): get_auth_request con 0x7f7c9006c900 auth_method 0 2024-05-15T13:20:34.896 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.892+0000 7f7c94fdc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:34.896 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.892+0000 7f7c94fdc640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:34.896 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.892+0000 7f7c94fdc640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:34.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.892+0000 7f7c8ffff640 10 monclient(hunting): get_auth_request con 0x7f7c900aeff0 auth_method 0 2024-05-15T13:20:34.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.892+0000 7f7c8ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:34.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.892+0000 7f7c8ffff640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:34.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.892+0000 7f7c8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:34.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.892+0000 7f7c957dd640 10 monclient(hunting): get_auth_request con 0x7f7c900af440 auth_method 0 2024-05-15T13:20:34.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.892+0000 7f7c957dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:34.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.892+0000 7f7c957dd640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:34.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.892+0000 7f7c957dd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:34.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.892+0000 7f7c94fdc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:34.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.892+0000 7f7c94fdc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:34.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.892+0000 7f7c94fdc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:34.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.892+0000 7f7c8ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:34.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.892+0000 7f7c8ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:34.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.892+0000 7f7c8ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:34.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.893+0000 7f7c957dd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:34.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.893+0000 7f7c957dd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:34.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.893+0000 7f7c957dd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:34.899 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.895+0000 7f7c94fdc640 10 monclient(hunting): handle_auth_done global_id 22228 payload 824 2024-05-15T13:20:34.899 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.895+0000 7f7c94fdc640 1 -- 172.21.15.117:0/3309012988 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7c900af440 msgr2=0x7f7c901cdaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:34.899 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.895+0000 7f7c94fdc640 1 --2- 172.21.15.117:0/3309012988 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7c900af440 0x7f7c901cdaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:34.899 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.895+0000 7f7c94fdc640 1 -- 172.21.15.117:0/3309012988 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7c900aeff0 msgr2=0x7f7c901cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:34.899 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.895+0000 7f7c94fdc640 1 --2- 172.21.15.117:0/3309012988 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7c900aeff0 0x7f7c901cd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:34.899 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.895+0000 7f7c94fdc640 10 monclient: _finish_hunting 0 2024-05-15T13:20:34.899 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.895+0000 7f7c94fdc640 1 monclient: found mon.noname-b 2024-05-15T13:20:34.899 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.895+0000 7f7c94fdc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:20:34.899 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.895+0000 7f7c94fdc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2024-05-15T13:20:34.899 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.895+0000 7f7c94fdc640 1 -- 172.21.15.117:0/3309012988 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c901cf6e0 con 0x7f7c9006c900 2024-05-15T13:20:34.899 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.897+0000 7f7c94fdc640 1 --2- 172.21.15.117:0/3309012988 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7c9006c900 0x7f7c900b5f30 secure :-1 s=READY pgs=5464 cs=0 l=1 rev1=1 crypto rx=0x7f7c840098c0 tx=0x7f7c840046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:34.899 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.897+0000 7f7c8d7fa640 10 client.?.objecter ms_handle_connect 0x7f7c9006c900 2024-05-15T13:20:34.899 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.897+0000 7f7c8d7fa640 10 client.?.objecter resend_mon_ops 2024-05-15T13:20:34.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.901+0000 7f7c8d7fa640 1 -- 172.21.15.117:0/3309012988 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c84032840 con 0x7f7c9006c900 2024-05-15T13:20:34.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.901+0000 7f7c8d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:34.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.901+0000 7f7c8d7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T13:20:34.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.901+0000 7f7c8d7fa640 10 monclient: dump: 2024-05-15T13:20:34.901 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:34.901 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:34.901 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:34.901 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:34.901 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:34.901 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:34.901 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:34.902 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:34.902 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:34.902 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:34.902 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.901+0000 7f7c8d7fa640 10 monclient: _finish_auth 0 2024-05-15T13:20:34.902 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.901+0000 7f7c8d7fa640 10 monclient: _check_auth_tickets 2024-05-15T13:20:34.902 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.901+0000 7f7c8d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:20:34.902 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.901+0000 7f7c8d7fa640 1 -- 172.21.15.117:0/3309012988 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c84032a00 con 0x7f7c9006c900 2024-05-15T13:20:34.902 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.901+0000 7f7c8d7fa640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:20:34.902 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.901+0000 7f7c8d7fa640 1 -- 172.21.15.117:0/3309012988 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c84032df0 con 0x7f7c9006c900 2024-05-15T13:20:34.902 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.901+0000 7f7c8d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:34.902 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.901+0000 7f7c8d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T13:20:34.902 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.901+0000 7f7c8d7fa640 10 monclient: dump: 2024-05-15T13:20:34.902 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:34.902 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:34.903 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:34.903 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:34.903 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:34.903 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:34.903 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:34.903 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:34.903 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:34.903 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:34.903 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.901+0000 7f7c97240640 5 monclient: authenticate success, global_id 22228 2024-05-15T13:20:34.903 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.901+0000 7f7c97240640 10 monclient: _renew_subs 2024-05-15T13:20:34.903 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.901+0000 7f7c97240640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T13:20:34.903 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.901+0000 7f7c97240640 1 -- 172.21.15.117:0/3309012988 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c901ce580 con 0x7f7c9006c900 2024-05-15T13:20:34.903 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.901+0000 7f7c8effd640 4 set_mon_vals no callback set 2024-05-15T13:20:34.903 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.901+0000 7f7c97240640 10 client.22228.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:20:34.903 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.901+0000 7f7c97240640 10 monclient: _renew_subs 2024-05-15T13:20:34.904 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.901+0000 7f7c97240640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T13:20:34.904 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.901+0000 7f7c97240640 1 -- 172.21.15.117:0/3309012988 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c901ce1a0 con 0x7f7c9006c900 2024-05-15T13:20:34.904 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.901+0000 7f7c97240640 1 librados: init done 2024-05-15T13:20:34.904 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.902+0000 7f7c97240640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T13:20:34.904 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.902+0000 7f7c97240640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:20:34.904 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.902+0000 7f7c97240640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2024-05-15T13:20:34.904 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:34.902+0000 7f7c97240640 1 -- 172.21.15.117:0/3309012988 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c901cf950 con 0x7f7c9006c900 2024-05-15T13:20:35.165 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:35.165+0000 7f7c8d7fa640 1 -- 172.21.15.117:0/3309012988 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f7c8404e430 con 0x7f7c9006c900 2024-05-15T13:20:35.165 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:35.166+0000 7f7c8d7fa640 1 --2- 172.21.15.117:0/3309012988 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f7c7406ceb0 0x7f7c7406f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:35.165 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:35.166+0000 7f7c8ffff640 1 --2- 172.21.15.117:0/3309012988 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f7c7406ceb0 0x7f7c7406f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:35.165 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:35.166+0000 7f7c8ffff640 10 monclient: get_auth_request con 0x7f7c7406ceb0 auth_method 0 2024-05-15T13:20:35.165 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:35.166+0000 7f7c8d7fa640 10 client.22228.objecter ms_handle_connect 0x7f7c7406ceb0 2024-05-15T13:20:35.165 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:35.166+0000 7f7c8ffff640 1 --2- 172.21.15.117:0/3309012988 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f7c7406ceb0 0x7f7c7406f370 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f7c9013ee80 tx=0x7f7c7c007e80 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:35.169 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:35.171+0000 7f7c8d7fa640 1 -- 172.21.15.117:0/3309012988 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(1225..1225 src has 912..1225) v4 ==== 6365+0+0 (secure 0 0 0) 0x7f7c8402f030 con 0x7f7c9006c900 2024-05-15T13:20:35.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:35.171+0000 7f7c8d7fa640 10 client.22228.objecter ms_dispatch 0x7f7c90000b90 osd_map(1225..1225 src has 912..1225) v4 2024-05-15T13:20:35.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:35.171+0000 7f7c8d7fa640 3 client.22228.objecter handle_osd_map got epochs [1225,1225] > 0 2024-05-15T13:20:35.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:35.171+0000 7f7c8d7fa640 3 client.22228.objecter handle_osd_map decoding full epoch 1225 2024-05-15T13:20:35.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:35.171+0000 7f7c8d7fa640 20 client.22228.objecter dump_active .. 0 homeless 2024-05-15T13:20:35.180 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:35.181+0000 7f7c8d7fa640 1 -- 172.21.15.117:0/3309012988 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f7c901cf950 con 0x7f7c9006c900 2024-05-15T13:20:35.180 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:35.181+0000 7f7c8d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:20:35.180 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:35.181+0000 7f7c8d7fa640 10 monclient: _finish_command 1 = system:0 2024-05-15T13:20:35.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:35.350+0000 7f7c97240640 1 -- 172.21.15.117:0/3309012988 --> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7c9004f0e0 con 0x7f7c7406ceb0 2024-05-15T13:20:35.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:35.353+0000 7f7c8d7fa640 1 -- 172.21.15.117:0/3309012988 <== mgr.4107 v2:172.21.15.117:6800/1785766523 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+115976 (secure 0 0 0) 0x7f7c9004f0e0 con 0x7f7c7406ceb0 2024-05-15T13:20:35.353 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T13:20:35.353 INFO:teuthology.orchestra.run.smithi117.stderr:dumped all 2024-05-15T13:20:35.355 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:35.355+0000 7f7c97240640 10 librados: watch_flush enter 2024-05-15T13:20:35.356 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:35.355+0000 7f7c97240640 10 librados: watch_flush exit 2024-05-15T13:20:35.356 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:35.355+0000 7f7c97240640 20 client.22228.objecter shutdown clearing up homeless session... 2024-05-15T13:20:35.356 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:35.355+0000 7f7c97240640 10 client.22228.objecter successfully canceled tick 2024-05-15T13:20:35.356 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:35.355+0000 7f7c97240640 1 -- 172.21.15.117:0/3309012988 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f7c7406ceb0 msgr2=0x7f7c7406f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:35.356 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:35.355+0000 7f7c97240640 1 --2- 172.21.15.117:0/3309012988 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f7c7406ceb0 0x7f7c7406f370 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f7c9013ee80 tx=0x7f7c7c007e80 comp rx=0 tx=0).stop 2024-05-15T13:20:35.356 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:35.355+0000 7f7c97240640 10 monclient: shutdown 2024-05-15T13:20:35.356 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:35.355+0000 7f7c97240640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:20:35.356 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:35.355+0000 7f7c97240640 1 -- 172.21.15.117:0/3309012988 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7c9006c900 msgr2=0x7f7c900b5f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:35.356 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:35.355+0000 7f7c97240640 1 --2- 172.21.15.117:0/3309012988 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7c9006c900 0x7f7c900b5f30 secure :-1 s=READY pgs=5464 cs=0 l=1 rev1=1 crypto rx=0x7f7c840098c0 tx=0x7f7c840046c0 comp rx=0 tx=0).stop 2024-05-15T13:20:35.356 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:35.356+0000 7f7c97240640 1 -- 172.21.15.117:0/3309012988 shutdown_connections 2024-05-15T13:20:35.356 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:35.356+0000 7f7c97240640 1 --2- 172.21.15.117:0/3309012988 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7c900af440 0x7f7c901cdaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:35.356 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:35.356+0000 7f7c97240640 1 --2- 172.21.15.117:0/3309012988 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7c900aeff0 0x7f7c901cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:35.356 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:35.356+0000 7f7c97240640 1 --2- 172.21.15.117:0/3309012988 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f7c7406ceb0 0x7f7c7406f370 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:35.356 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:35.356+0000 7f7c97240640 1 --2- 172.21.15.117:0/3309012988 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7c9006c900 0x7f7c900b5f30 unknown :-1 s=CLOSED pgs=5464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:35.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:35.356+0000 7f7c97240640 1 -- 172.21.15.117:0/3309012988 >> 172.21.15.117:0/3309012988 conn(0x7f7c900b2ac0 msgr2=0x7f7c90145250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:20:35.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:35.356+0000 7f7c97240640 1 -- 172.21.15.117:0/3309012988 shutdown_connections 2024-05-15T13:20:35.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:35.356+0000 7f7c97240640 1 -- 172.21.15.117:0/3309012988 wait complete. 2024-05-15T13:20:35.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:35.356+0000 7f7c97240640 1 librados: shutdown 2024-05-15T13:20:35.365 INFO:teuthology.orchestra.run.smithi117.stdout:{"pg_ready":true,"pg_map":{"version":2361,"stamp":"2024-05-15T13:20:34.614825+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85,"ondisk_log_size":85,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":602548,"kb_used_data":42200,"kb_used_omap":12,"kb_used_meta":560243,"kb_avail":754372172,"statfs":{"total":773094113280,"available":772477104128,"internally_reserved":0,"allocated":43212800,"data_stored":21573962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":573689394},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":60,"apply_latency_ms":60,"commit_latency_ns":60000000,"apply_latency_ns":60000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002232"},"pg_stats":[{"pgid":"6.16","version":"0'0","reported_seq":2364,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:26.892883+0000","last_change":"2024-05-15T12:44:59.567389+0000","last_active":"2024-05-15T13:20:26.892883+0000","last_peered":"2024-05-15T13:20:26.892883+0000","last_clean":"2024-05-15T13:20:26.892883+0000","last_became_active":"2024-05-15T12:44:59.550282+0000","last_became_peered":"2024-05-15T12:44:59.550282+0000","last_unstale":"2024-05-15T13:20:26.892883+0000","last_undegraded":"2024-05-15T13:20:26.892883+0000","last_fullsized":"2024-05-15T13:20:26.892883+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:44:22.143842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":2380,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:26.918783+0000","last_change":"2024-05-15T12:44:51.243387+0000","last_active":"2024-05-15T13:20:26.918783+0000","last_peered":"2024-05-15T13:20:26.918783+0000","last_clean":"2024-05-15T13:20:26.918783+0000","last_became_active":"2024-05-15T12:44:51.202412+0000","last_became_peered":"2024-05-15T12:44:51.202412+0000","last_unstale":"2024-05-15T13:20:26.918783+0000","last_undegraded":"2024-05-15T13:20:26.918783+0000","last_fullsized":"2024-05-15T13:20:26.918783+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":2365,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:26.886583+0000","last_change":"2024-05-15T12:45:13.532350+0000","last_active":"2024-05-15T13:20:26.886583+0000","last_peered":"2024-05-15T13:20:26.886583+0000","last_clean":"2024-05-15T13:20:26.886583+0000","last_became_active":"2024-05-15T12:45:11.468881+0000","last_became_peered":"2024-05-15T12:45:11.468881+0000","last_unstale":"2024-05-15T13:20:26.886583+0000","last_undegraded":"2024-05-15T13:20:26.886583+0000","last_fullsized":"2024-05-15T13:20:26.886583+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T15:53:17.739425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":2366,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:27.570746+0000","last_change":"2024-05-15T12:44:55.797152+0000","last_active":"2024-05-15T13:20:27.570746+0000","last_peered":"2024-05-15T13:20:27.570746+0000","last_clean":"2024-05-15T13:20:27.570746+0000","last_became_active":"2024-05-15T12:44:55.775514+0000","last_became_peered":"2024-05-15T12:44:55.775514+0000","last_unstale":"2024-05-15T13:20:27.570746+0000","last_undegraded":"2024-05-15T13:20:27.570746+0000","last_fullsized":"2024-05-15T13:20:27.570746+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:09:17.199545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":2365,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:27.486800+0000","last_change":"2024-05-15T12:44:57.700224+0000","last_active":"2024-05-15T13:20:27.486800+0000","last_peered":"2024-05-15T13:20:27.486800+0000","last_clean":"2024-05-15T13:20:27.486800+0000","last_became_active":"2024-05-15T12:44:57.673614+0000","last_became_peered":"2024-05-15T12:44:57.673614+0000","last_unstale":"2024-05-15T13:20:27.486800+0000","last_undegraded":"2024-05-15T13:20:27.486800+0000","last_fullsized":"2024-05-15T13:20:27.486800+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":2365,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:26.883191+0000","last_change":"2024-05-15T12:44:59.486234+0000","last_active":"2024-05-15T13:20:26.883191+0000","last_peered":"2024-05-15T13:20:26.883191+0000","last_clean":"2024-05-15T13:20:26.883191+0000","last_became_active":"2024-05-15T12:44:59.480451+0000","last_became_peered":"2024-05-15T12:44:59.480451+0000","last_unstale":"2024-05-15T13:20:26.883191+0000","last_undegraded":"2024-05-15T13:20:26.883191+0000","last_fullsized":"2024-05-15T13:20:26.883191+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:28:22.234323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":2369,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:26.899621+0000","last_change":"2024-05-15T12:44:54.924153+0000","last_active":"2024-05-15T13:20:26.899621+0000","last_peered":"2024-05-15T13:20:26.899621+0000","last_clean":"2024-05-15T13:20:26.899621+0000","last_became_active":"2024-05-15T12:44:54.914910+0000","last_became_peered":"2024-05-15T12:44:54.914910+0000","last_unstale":"2024-05-15T13:20:26.899621+0000","last_undegraded":"2024-05-15T13:20:26.899621+0000","last_fullsized":"2024-05-15T13:20:26.899621+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-17T00:36:49.391069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":2366,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:26.881296+0000","last_change":"2024-05-15T12:44:59.650327+0000","last_active":"2024-05-15T13:20:26.881296+0000","last_peered":"2024-05-15T13:20:26.881296+0000","last_clean":"2024-05-15T13:20:26.881296+0000","last_became_active":"2024-05-15T12:44:59.595062+0000","last_became_peered":"2024-05-15T12:44:59.595062+0000","last_unstale":"2024-05-15T13:20:26.881296+0000","last_undegraded":"2024-05-15T13:20:26.881296+0000","last_fullsized":"2024-05-15T13:20:26.881296+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T12:52:31.383828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":2382,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:26.912985+0000","last_change":"2024-05-15T12:44:51.802252+0000","last_active":"2024-05-15T13:20:26.912985+0000","last_peered":"2024-05-15T13:20:26.912985+0000","last_clean":"2024-05-15T13:20:26.912985+0000","last_became_active":"2024-05-15T12:44:51.785998+0000","last_became_peered":"2024-05-15T12:44:51.785998+0000","last_unstale":"2024-05-15T13:20:26.912985+0000","last_undegraded":"2024-05-15T13:20:26.912985+0000","last_fullsized":"2024-05-15T13:20:26.912985+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":2379,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:28.093073+0000","last_change":"2024-05-15T12:44:53.614201+0000","last_active":"2024-05-15T13:20:28.093073+0000","last_peered":"2024-05-15T13:20:28.093073+0000","last_clean":"2024-05-15T13:20:28.093073+0000","last_became_active":"2024-05-15T12:44:53.608407+0000","last_became_peered":"2024-05-15T12:44:53.608407+0000","last_unstale":"2024-05-15T13:20:28.093073+0000","last_undegraded":"2024-05-15T13:20:28.093073+0000","last_fullsized":"2024-05-15T13:20:28.093073+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:45:19.480038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":2367,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:28.106948+0000","last_change":"2024-05-15T12:44:52.221672+0000","last_active":"2024-05-15T13:20:28.106948+0000","last_peered":"2024-05-15T13:20:28.106948+0000","last_clean":"2024-05-15T13:20:28.106948+0000","last_became_active":"2024-05-15T12:44:52.212480+0000","last_became_peered":"2024-05-15T12:44:52.212480+0000","last_unstale":"2024-05-15T13:20:28.106948+0000","last_undegraded":"2024-05-15T13:20:28.106948+0000","last_fullsized":"2024-05-15T13:20:28.106948+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:41:34.164739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":2382,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:26.907528+0000","last_change":"2024-05-15T12:44:56.446419+0000","last_active":"2024-05-15T13:20:26.907528+0000","last_peered":"2024-05-15T13:20:26.907528+0000","last_clean":"2024-05-15T13:20:26.907528+0000","last_became_active":"2024-05-15T12:44:56.197837+0000","last_became_peered":"2024-05-15T12:44:56.197837+0000","last_unstale":"2024-05-15T13:20:26.907528+0000","last_undegraded":"2024-05-15T13:20:26.907528+0000","last_fullsized":"2024-05-15T13:20:26.907528+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":2368,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:26.935173+0000","last_change":"2024-05-15T12:44:53.399230+0000","last_active":"2024-05-15T13:20:26.935173+0000","last_peered":"2024-05-15T13:20:26.935173+0000","last_clean":"2024-05-15T13:20:26.935173+0000","last_became_active":"2024-05-15T12:44:53.377773+0000","last_became_peered":"2024-05-15T12:44:53.377773+0000","last_unstale":"2024-05-15T13:20:26.935173+0000","last_undegraded":"2024-05-15T13:20:26.935173+0000","last_fullsized":"2024-05-15T13:20:26.935173+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:12:39.424987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"1225'78","reported_seq":2414,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:29.968940+0000","last_change":"2024-05-15T13:20:29.970241+0000","last_active":"2024-05-15T13:20:29.968940+0000","last_peered":"2024-05-15T13:20:29.968940+0000","last_clean":"2024-05-15T13:20:29.968940+0000","last_became_active":"2024-05-15T12:43:04.148735+0000","last_became_peered":"2024-05-15T12:43:04.148735+0000","last_unstale":"2024-05-15T13:20:29.968940+0000","last_undegraded":"2024-05-15T13:20:29.968940+0000","last_fullsized":"2024-05-15T13:20:29.968940+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"1225'78","last_scrub_stamp":"2024-05-15T13:20:29.967300+0000","last_deep_scrub":"1225'78","last_deep_scrub_stamp":"2024-05-15T13:20:29.967300+0000","last_clean_scrub_stamp":"2024-05-15T13:20:29.967300+0000","objects_scrubbed":2,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T16:05:45.963490+0000","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2378,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:26.924387+0000","last_change":"2024-05-15T12:43:09.453018+0000","last_active":"2024-05-15T13:20:26.924387+0000","last_peered":"2024-05-15T13:20:26.924387+0000","last_clean":"2024-05-15T13:20:26.924387+0000","last_became_active":"2024-05-15T12:43:09.445878+0000","last_became_peered":"2024-05-15T12:43:09.445878+0000","last_unstale":"2024-05-15T13:20:26.924387+0000","last_undegraded":"2024-05-15T13:20:26.924387+0000","last_fullsized":"2024-05-15T13:20:26.924387+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_clean_scrub_stamp":"2024-05-15T12:43:07.133352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:55:05.766298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":2374,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:27.587342+0000","last_change":"2024-05-15T12:44:50.705660+0000","last_active":"2024-05-15T13:20:27.587342+0000","last_peered":"2024-05-15T13:20:27.587342+0000","last_clean":"2024-05-15T13:20:27.587342+0000","last_became_active":"2024-05-15T12:44:50.696664+0000","last_became_peered":"2024-05-15T12:44:50.696664+0000","last_unstale":"2024-05-15T13:20:27.587342+0000","last_undegraded":"2024-05-15T13:20:27.587342+0000","last_fullsized":"2024-05-15T13:20:27.587342+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:15:31.819701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2379,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:26.896447+0000","last_change":"2024-05-15T12:43:08.704425+0000","last_active":"2024-05-15T13:20:26.896447+0000","last_peered":"2024-05-15T13:20:26.896447+0000","last_clean":"2024-05-15T13:20:26.896447+0000","last_became_active":"2024-05-15T12:43:08.677751+0000","last_became_peered":"2024-05-15T12:43:08.677751+0000","last_unstale":"2024-05-15T13:20:26.896447+0000","last_undegraded":"2024-05-15T13:20:26.896447+0000","last_fullsized":"2024-05-15T13:20:26.896447+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_clean_scrub_stamp":"2024-05-15T12:43:07.133352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-17T00:22:51.843061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":2373,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:29.500116+0000","last_change":"2024-05-15T13:20:29.500929+0000","last_active":"2024-05-15T13:20:29.500116+0000","last_peered":"2024-05-15T13:20:29.500116+0000","last_clean":"2024-05-15T13:20:29.500116+0000","last_became_active":"2024-05-15T12:44:57.552111+0000","last_became_peered":"2024-05-15T12:44:57.552111+0000","last_unstale":"2024-05-15T13:20:29.500116+0000","last_undegraded":"2024-05-15T13:20:29.500116+0000","last_fullsized":"2024-05-15T13:20:29.500116+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:20:29.495828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:20:29.495828+0000","last_clean_scrub_stamp":"2024-05-15T13:20:29.495828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:57:37.144480+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2382,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:27.582027+0000","last_change":"2024-05-15T12:43:08.634999+0000","last_active":"2024-05-15T13:20:27.582027+0000","last_peered":"2024-05-15T13:20:27.582027+0000","last_clean":"2024-05-15T13:20:27.582027+0000","last_became_active":"2024-05-15T12:43:08.600745+0000","last_became_peered":"2024-05-15T12:43:08.600745+0000","last_unstale":"2024-05-15T13:20:27.582027+0000","last_undegraded":"2024-05-15T13:20:27.582027+0000","last_fullsized":"2024-05-15T13:20:27.582027+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_clean_scrub_stamp":"2024-05-15T12:43:07.133352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:17:56.643544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":2380,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:28.083145+0000","last_change":"2024-05-15T12:44:51.361084+0000","last_active":"2024-05-15T13:20:28.083145+0000","last_peered":"2024-05-15T13:20:28.083145+0000","last_clean":"2024-05-15T13:20:28.083145+0000","last_became_active":"2024-05-15T12:44:51.354879+0000","last_became_peered":"2024-05-15T12:44:51.354879+0000","last_unstale":"2024-05-15T13:20:28.083145+0000","last_undegraded":"2024-05-15T13:20:28.083145+0000","last_fullsized":"2024-05-15T13:20:28.083145+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:53:33.707185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2378,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:26.906310+0000","last_change":"2024-05-15T12:43:09.375465+0000","last_active":"2024-05-15T13:20:26.906310+0000","last_peered":"2024-05-15T13:20:26.906310+0000","last_clean":"2024-05-15T13:20:26.906310+0000","last_became_active":"2024-05-15T12:43:09.363876+0000","last_became_peered":"2024-05-15T12:43:09.363876+0000","last_unstale":"2024-05-15T13:20:26.906310+0000","last_undegraded":"2024-05-15T13:20:26.906310+0000","last_fullsized":"2024-05-15T13:20:26.906310+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_clean_scrub_stamp":"2024-05-15T12:43:07.133352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:27:46.599385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":2374,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:27.576906+0000","last_change":"2024-05-15T12:44:54.188435+0000","last_active":"2024-05-15T13:20:27.576906+0000","last_peered":"2024-05-15T13:20:27.576906+0000","last_clean":"2024-05-15T13:20:27.576906+0000","last_became_active":"2024-05-15T12:44:54.179460+0000","last_became_peered":"2024-05-15T12:44:54.179460+0000","last_unstale":"2024-05-15T13:20:27.576906+0000","last_undegraded":"2024-05-15T13:20:27.576906+0000","last_fullsized":"2024-05-15T13:20:27.576906+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T12:52:27.875820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2389,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:28.080772+0000","last_change":"2024-05-15T12:43:09.384765+0000","last_active":"2024-05-15T13:20:28.080772+0000","last_peered":"2024-05-15T13:20:28.080772+0000","last_clean":"2024-05-15T13:20:28.080772+0000","last_became_active":"2024-05-15T12:43:09.377725+0000","last_became_peered":"2024-05-15T12:43:09.377725+0000","last_unstale":"2024-05-15T13:20:28.080772+0000","last_undegraded":"2024-05-15T13:20:28.080772+0000","last_fullsized":"2024-05-15T13:20:28.080772+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_clean_scrub_stamp":"2024-05-15T12:43:07.133352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:57:59.671602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"47'7","reported_seq":2382,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:28.086237+0000","last_change":"2024-05-15T12:44:58.436179+0000","last_active":"2024-05-15T13:20:28.086237+0000","last_peered":"2024-05-15T13:20:28.086237+0000","last_clean":"2024-05-15T13:20:28.086237+0000","last_became_active":"2024-05-15T12:44:58.318306+0000","last_became_peered":"2024-05-15T12:44:58.318306+0000","last_unstale":"2024-05-15T13:20:28.086237+0000","last_undegraded":"2024-05-15T13:20:28.086237+0000","last_fullsized":"2024-05-15T13:20:28.086237+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:59:46.310559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2378,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:26.930732+0000","last_change":"2024-05-15T12:43:08.778833+0000","last_active":"2024-05-15T13:20:26.930732+0000","last_peered":"2024-05-15T13:20:26.930732+0000","last_clean":"2024-05-15T13:20:26.930732+0000","last_became_active":"2024-05-15T12:43:08.773614+0000","last_became_peered":"2024-05-15T12:43:08.773614+0000","last_unstale":"2024-05-15T13:20:26.930732+0000","last_undegraded":"2024-05-15T13:20:26.930732+0000","last_fullsized":"2024-05-15T13:20:26.930732+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_clean_scrub_stamp":"2024-05-15T12:43:07.133352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:22:59.785687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":2370,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:26.918589+0000","last_change":"2024-05-15T12:45:08.669407+0000","last_active":"2024-05-15T13:20:26.918589+0000","last_peered":"2024-05-15T13:20:26.918589+0000","last_clean":"2024-05-15T13:20:26.918589+0000","last_became_active":"2024-05-15T12:45:08.661140+0000","last_became_peered":"2024-05-15T12:45:08.661140+0000","last_unstale":"2024-05-15T13:20:26.918589+0000","last_undegraded":"2024-05-15T13:20:26.918589+0000","last_fullsized":"2024-05-15T13:20:26.918589+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:55:43.700115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2376,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:26.894733+0000","last_change":"2024-05-15T12:43:09.478046+0000","last_active":"2024-05-15T13:20:26.894733+0000","last_peered":"2024-05-15T13:20:26.894733+0000","last_clean":"2024-05-15T13:20:26.894733+0000","last_became_active":"2024-05-15T12:43:09.448237+0000","last_became_peered":"2024-05-15T12:43:09.448237+0000","last_unstale":"2024-05-15T13:20:26.894733+0000","last_undegraded":"2024-05-15T13:20:26.894733+0000","last_fullsized":"2024-05-15T13:20:26.894733+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_clean_scrub_stamp":"2024-05-15T12:43:07.133352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:11:17.055288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":2365,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:27.497756+0000","last_change":"2024-05-15T12:45:18.657708+0000","last_active":"2024-05-15T13:20:27.497756+0000","last_peered":"2024-05-15T13:20:27.497756+0000","last_clean":"2024-05-15T13:20:27.497756+0000","last_became_active":"2024-05-15T12:45:18.642568+0000","last_became_peered":"2024-05-15T12:45:18.642568+0000","last_unstale":"2024-05-15T13:20:27.497756+0000","last_undegraded":"2024-05-15T13:20:27.497756+0000","last_fullsized":"2024-05-15T13:20:27.497756+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2377,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:26.937343+0000","last_change":"2024-05-15T12:43:09.429828+0000","last_active":"2024-05-15T13:20:26.937343+0000","last_peered":"2024-05-15T13:20:26.937343+0000","last_clean":"2024-05-15T13:20:26.937343+0000","last_became_active":"2024-05-15T12:43:09.424460+0000","last_became_peered":"2024-05-15T12:43:09.424460+0000","last_unstale":"2024-05-15T13:20:26.937343+0000","last_undegraded":"2024-05-15T13:20:26.937343+0000","last_fullsized":"2024-05-15T13:20:26.937343+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:43:07.133352+0000","last_clean_scrub_stamp":"2024-05-15T12:43:07.133352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:12:21.424590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":2367,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:28.084278+0000","last_change":"2024-05-15T12:44:52.129096+0000","last_active":"2024-05-15T13:20:28.084278+0000","last_peered":"2024-05-15T13:20:28.084278+0000","last_clean":"2024-05-15T13:20:28.084278+0000","last_became_active":"2024-05-15T12:44:52.088745+0000","last_became_peered":"2024-05-15T12:44:52.088745+0000","last_unstale":"2024-05-15T13:20:28.084278+0000","last_undegraded":"2024-05-15T13:20:28.084278+0000","last_fullsized":"2024-05-15T13:20:28.084278+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:16:34.966002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":2365,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:26.872600+0000","last_change":"2024-05-15T12:44:56.480904+0000","last_active":"2024-05-15T13:20:26.872600+0000","last_peered":"2024-05-15T13:20:26.872600+0000","last_clean":"2024-05-15T13:20:26.872600+0000","last_became_active":"2024-05-15T12:44:56.470246+0000","last_became_peered":"2024-05-15T12:44:56.470246+0000","last_unstale":"2024-05-15T13:20:26.872600+0000","last_undegraded":"2024-05-15T13:20:26.872600+0000","last_fullsized":"2024-05-15T13:20:26.872600+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:53:59.488101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":2371,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:27.589292+0000","last_change":"2024-05-15T12:44:51.677390+0000","last_active":"2024-05-15T13:20:27.589292+0000","last_peered":"2024-05-15T13:20:27.589292+0000","last_clean":"2024-05-15T13:20:27.589292+0000","last_became_active":"2024-05-15T12:44:51.660527+0000","last_became_peered":"2024-05-15T12:44:51.660527+0000","last_unstale":"2024-05-15T13:20:27.589292+0000","last_undegraded":"2024-05-15T13:20:27.589292+0000","last_fullsized":"2024-05-15T13:20:27.589292+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:25:49.920807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":2364,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:27.488708+0000","last_change":"2024-05-15T12:45:12.010393+0000","last_active":"2024-05-15T13:20:27.488708+0000","last_peered":"2024-05-15T13:20:27.488708+0000","last_clean":"2024-05-15T13:20:27.488708+0000","last_became_active":"2024-05-15T12:45:11.998256+0000","last_became_peered":"2024-05-15T12:45:11.998256+0000","last_unstale":"2024-05-15T13:20:27.488708+0000","last_undegraded":"2024-05-15T13:20:27.488708+0000","last_fullsized":"2024-05-15T13:20:27.488708+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":2365,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:28.099898+0000","last_change":"2024-05-15T12:44:55.564951+0000","last_active":"2024-05-15T13:20:28.099898+0000","last_peered":"2024-05-15T13:20:28.099898+0000","last_clean":"2024-05-15T13:20:28.099898+0000","last_became_active":"2024-05-15T12:44:55.557214+0000","last_became_peered":"2024-05-15T12:44:55.557214+0000","last_unstale":"2024-05-15T13:20:28.099898+0000","last_undegraded":"2024-05-15T13:20:28.099898+0000","last_fullsized":"2024-05-15T13:20:28.099898+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:53:15.834038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":2367,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:26.922438+0000","last_change":"2024-05-15T12:44:56.078343+0000","last_active":"2024-05-15T13:20:26.922438+0000","last_peered":"2024-05-15T13:20:26.922438+0000","last_clean":"2024-05-15T13:20:26.922438+0000","last_became_active":"2024-05-15T12:44:56.072618+0000","last_became_peered":"2024-05-15T12:44:56.072618+0000","last_unstale":"2024-05-15T13:20:26.922438+0000","last_undegraded":"2024-05-15T13:20:26.922438+0000","last_fullsized":"2024-05-15T13:20:26.922438+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:22:00.980595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":2369,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:28.102719+0000","last_change":"2024-05-15T12:44:55.746501+0000","last_active":"2024-05-15T13:20:28.102719+0000","last_peered":"2024-05-15T13:20:28.102719+0000","last_clean":"2024-05-15T13:20:28.102719+0000","last_became_active":"2024-05-15T12:44:55.739811+0000","last_became_peered":"2024-05-15T12:44:55.739811+0000","last_unstale":"2024-05-15T13:20:28.102719+0000","last_undegraded":"2024-05-15T13:20:28.102719+0000","last_fullsized":"2024-05-15T13:20:28.102719+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T12:49:40.051745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":2369,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:26.939363+0000","last_change":"2024-05-15T12:44:56.495925+0000","last_active":"2024-05-15T13:20:26.939363+0000","last_peered":"2024-05-15T13:20:26.939363+0000","last_clean":"2024-05-15T13:20:26.939363+0000","last_became_active":"2024-05-15T12:44:56.429996+0000","last_became_peered":"2024-05-15T12:44:56.429996+0000","last_unstale":"2024-05-15T13:20:26.939363+0000","last_undegraded":"2024-05-15T13:20:26.939363+0000","last_fullsized":"2024-05-15T13:20:26.939363+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:18:31.842665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":2363,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:27.481445+0000","last_change":"2024-05-15T12:45:18.556824+0000","last_active":"2024-05-15T13:20:27.481445+0000","last_peered":"2024-05-15T13:20:27.481445+0000","last_clean":"2024-05-15T13:20:27.481445+0000","last_became_active":"2024-05-15T12:45:18.546529+0000","last_became_peered":"2024-05-15T12:45:18.546529+0000","last_unstale":"2024-05-15T13:20:27.481445+0000","last_undegraded":"2024-05-15T13:20:27.481445+0000","last_fullsized":"2024-05-15T13:20:27.481445+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":2364,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:26.874399+0000","last_change":"2024-05-15T12:44:56.642476+0000","last_active":"2024-05-15T13:20:26.874399+0000","last_peered":"2024-05-15T13:20:26.874399+0000","last_clean":"2024-05-15T13:20:26.874399+0000","last_became_active":"2024-05-15T12:44:56.634257+0000","last_became_peered":"2024-05-15T12:44:56.634257+0000","last_unstale":"2024-05-15T13:20:26.874399+0000","last_undegraded":"2024-05-15T13:20:26.874399+0000","last_fullsized":"2024-05-15T13:20:26.874399+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T15:44:34.817095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":2366,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:28.093257+0000","last_change":"2024-05-15T12:44:52.504602+0000","last_active":"2024-05-15T13:20:28.093257+0000","last_peered":"2024-05-15T13:20:28.093257+0000","last_clean":"2024-05-15T13:20:28.093257+0000","last_became_active":"2024-05-15T12:44:52.483168+0000","last_became_peered":"2024-05-15T12:44:52.483168+0000","last_unstale":"2024-05-15T13:20:28.093257+0000","last_undegraded":"2024-05-15T13:20:28.093257+0000","last_fullsized":"2024-05-15T13:20:28.093257+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:17:24.835749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":2367,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:26.913258+0000","last_change":"2024-05-15T12:45:12.195717+0000","last_active":"2024-05-15T13:20:26.913258+0000","last_peered":"2024-05-15T13:20:26.913258+0000","last_clean":"2024-05-15T13:20:26.913258+0000","last_became_active":"2024-05-15T12:45:12.190202+0000","last_became_peered":"2024-05-15T12:45:12.190202+0000","last_unstale":"2024-05-15T13:20:26.913258+0000","last_undegraded":"2024-05-15T13:20:26.913258+0000","last_fullsized":"2024-05-15T13:20:26.913258+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:44:46.163820+0000","last_clean_scrub_stamp":"2024-05-15T12:44:46.163820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:05:34.894666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":20,"seq":85899346375,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72844,"kb_used_data":5568,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94298996,"statfs":{"total":96636764160,"available":96562171904,"internally_reserved":0,"allocated":5701632,"data_stored":2995124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":0,"up_from":20,"seq":85899346374,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68292,"kb_used_data":5176,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303548,"statfs":{"total":96636764160,"available":96566833152,"internally_reserved":0,"allocated":5300224,"data_stored":2597284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[]},{"osd":2,"up_from":20,"seq":85899346374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94596,"kb_used_data":5176,"kb_used_omap":1,"kb_used_meta":89406,"kb_avail":94277244,"statfs":{"total":96636764160,"available":96539897856,"internally_reserved":0,"allocated":5300224,"data_stored":2597284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":91552198},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[]},{"osd":6,"up_from":19,"seq":81604379078,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72456,"kb_used_data":5180,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299384,"statfs":{"total":96636764160,"available":96562569216,"internally_reserved":0,"allocated":5304320,"data_stored":2597289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":19,"seq":81604379078,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72452,"kb_used_data":5176,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299388,"statfs":{"total":96636764160,"available":96562573312,"internally_reserved":0,"allocated":5300224,"data_stored":2597284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509892,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72456,"kb_used_data":5180,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299384,"statfs":{"total":96636764160,"available":96562569216,"internally_reserved":0,"allocated":5304320,"data_stored":2597289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542597,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72840,"kb_used_data":5568,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299000,"statfs":{"total":96636764160,"available":96562176000,"internally_reserved":0,"allocated":5701632,"data_stored":2995124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":7,"up_from":13,"seq":55834575302,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76612,"kb_used_data":5176,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94295228,"statfs":{"total":96636764160,"available":96558313472,"internally_reserved":0,"allocated":5300224,"data_stored":2597284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T13:20:35.370 INFO:tasks.ceph:pgid 6.16 last_scrub_stamp 2024-05-15T12:44:46.163820+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=44, tm_sec=46, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=20, tm_sec=27, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:20:35.371 INFO:tasks.ceph:pgid 6.17 last_scrub_stamp 2024-05-15T12:44:46.163820+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=44, tm_sec=46, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=20, tm_sec=27, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:20:35.371 INFO:tasks.ceph:pgid 6.14 last_scrub_stamp 2024-05-15T12:44:46.163820+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=44, tm_sec=46, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=20, tm_sec=27, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:20:35.371 INFO:tasks.ceph:pgid 6.15 last_scrub_stamp 2024-05-15T12:44:46.163820+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=44, tm_sec=46, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=20, tm_sec=27, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:20:35.371 INFO:tasks.ceph:pgid 6.a last_scrub_stamp 2024-05-15T12:44:46.163820+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=44, tm_sec=46, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=20, tm_sec=27, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:20:35.371 INFO:tasks.ceph:pgid 6.b last_scrub_stamp 2024-05-15T12:44:46.163820+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=44, tm_sec=46, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=20, tm_sec=27, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:20:35.371 INFO:tasks.ceph:pgid 6.8 last_scrub_stamp 2024-05-15T12:44:46.163820+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=44, tm_sec=46, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=20, tm_sec=27, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:20:35.371 INFO:tasks.ceph:pgid 6.9 last_scrub_stamp 2024-05-15T12:44:46.163820+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=44, tm_sec=46, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=20, tm_sec=27, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:20:35.371 INFO:tasks.ceph:pgid 6.e last_scrub_stamp 2024-05-15T12:44:46.163820+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=44, tm_sec=46, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=20, tm_sec=27, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:20:35.371 INFO:tasks.ceph:pgid 6.f last_scrub_stamp 2024-05-15T12:44:46.163820+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=44, tm_sec=46, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=20, tm_sec=27, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:20:35.371 INFO:tasks.ceph:pgid 6.c last_scrub_stamp 2024-05-15T12:44:46.163820+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=44, tm_sec=46, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=20, tm_sec=27, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:20:35.372 INFO:tasks.ceph:pgid 6.d last_scrub_stamp 2024-05-15T12:44:46.163820+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=44, tm_sec=46, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=20, tm_sec=27, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:20:35.372 INFO:tasks.ceph:pgid 6.7 last_scrub_stamp 2024-05-15T12:44:46.163820+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=44, tm_sec=46, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=20, tm_sec=27, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:20:35.372 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-05-15T12:43:07.133352+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=43, tm_sec=7, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=20, tm_sec=27, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:20:35.372 INFO:tasks.ceph:pgid 6.4 last_scrub_stamp 2024-05-15T12:44:46.163820+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=44, tm_sec=46, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=20, tm_sec=27, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:20:35.372 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-05-15T12:43:07.133352+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=43, tm_sec=7, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=20, tm_sec=27, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:20:35.372 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-05-15T12:43:07.133352+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=43, tm_sec=7, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=20, tm_sec=27, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:20:35.372 INFO:tasks.ceph:pgid 6.6 last_scrub_stamp 2024-05-15T12:44:46.163820+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=44, tm_sec=46, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=20, tm_sec=27, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:20:35.372 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-05-15T12:43:07.133352+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=43, tm_sec=7, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=20, tm_sec=27, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:20:35.372 INFO:tasks.ceph:pgid 6.0 last_scrub_stamp 2024-05-15T12:44:46.163820+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=44, tm_sec=46, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=20, tm_sec=27, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:20:35.373 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-05-15T12:43:07.133352+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=43, tm_sec=7, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=20, tm_sec=27, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:20:35.373 INFO:tasks.ceph:pgid 6.1 last_scrub_stamp 2024-05-15T12:44:46.163820+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=44, tm_sec=46, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=20, tm_sec=27, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:20:35.373 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-05-15T12:43:07.133352+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=43, tm_sec=7, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=20, tm_sec=27, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:20:35.373 INFO:tasks.ceph:pgid 6.2 last_scrub_stamp 2024-05-15T12:44:46.163820+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=44, tm_sec=46, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=20, tm_sec=27, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:20:35.373 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-05-15T12:43:07.133352+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=43, tm_sec=7, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=20, tm_sec=27, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:20:35.373 INFO:tasks.ceph:pgid 6.3 last_scrub_stamp 2024-05-15T12:44:46.163820+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=44, tm_sec=46, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=20, tm_sec=27, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:20:35.373 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-05-15T12:43:07.133352+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=43, tm_sec=7, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=20, tm_sec=27, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:20:35.373 INFO:tasks.ceph:pgid 6.11 last_scrub_stamp 2024-05-15T12:44:46.163820+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=44, tm_sec=46, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=20, tm_sec=27, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:20:35.373 INFO:tasks.ceph:pgid 6.10 last_scrub_stamp 2024-05-15T12:44:46.163820+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=44, tm_sec=46, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=20, tm_sec=27, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:20:35.373 INFO:tasks.ceph:pgid 6.13 last_scrub_stamp 2024-05-15T12:44:46.163820+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=44, tm_sec=46, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=20, tm_sec=27, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:20:35.374 INFO:tasks.ceph:pgid 6.12 last_scrub_stamp 2024-05-15T12:44:46.163820+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=44, tm_sec=46, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=20, tm_sec=27, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:20:35.374 INFO:tasks.ceph:pgid 6.1d last_scrub_stamp 2024-05-15T12:44:46.163820+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=44, tm_sec=46, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=20, tm_sec=27, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:20:35.374 INFO:tasks.ceph:pgid 6.1c last_scrub_stamp 2024-05-15T12:44:46.163820+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=44, tm_sec=46, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=20, tm_sec=27, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:20:35.374 INFO:tasks.ceph:pgid 6.1f last_scrub_stamp 2024-05-15T12:44:46.163820+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=44, tm_sec=46, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=20, tm_sec=27, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:20:35.374 INFO:tasks.ceph:pgid 6.1e last_scrub_stamp 2024-05-15T12:44:46.163820+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=44, tm_sec=46, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=20, tm_sec=27, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:20:35.374 INFO:tasks.ceph:pgid 6.19 last_scrub_stamp 2024-05-15T12:44:46.163820+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=44, tm_sec=46, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=20, tm_sec=27, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:20:35.374 INFO:tasks.ceph:pgid 6.18 last_scrub_stamp 2024-05-15T12:44:46.163820+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=44, tm_sec=46, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=20, tm_sec=27, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:20:35.374 INFO:tasks.ceph:pgid 6.1b last_scrub_stamp 2024-05-15T12:44:46.163820+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=44, tm_sec=46, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=20, tm_sec=27, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:20:35.374 INFO:tasks.ceph:pgid 6.1a last_scrub_stamp 2024-05-15T12:44:46.163820+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=44, tm_sec=46, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=20, tm_sec=27, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:20:35.375 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-05-15T13:20:36.960 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:38:44.130 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:20:36.962 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:38:44.132 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:20:36.978 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:38:44.148 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:20:37.007 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:38:44.170 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:20:37.009 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:38:44.172 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:20:37.015 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:38:44.185 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:20:37.017 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:38:44.187 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:20:37.023 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:38:44.186 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:20:37.032 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:38:44.202 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:20:41.971 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:38:49.141 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:20:41.973 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:38:49.143 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:20:41.984 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:38:49.154 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:20:42.019 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:38:49.182 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:20:42.020 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:38:49.183 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:20:42.024 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:38:49.193 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:20:42.030 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:38:49.200 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:20:42.030 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:38:49.193 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:20:42.040 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:38:49.210 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:20:55.376 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-15T13:20:55.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.487+0000 7f03abdfe640 10 monclient: get_monmap_and_config 2024-05-15T13:20:55.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.487+0000 7f03abdfe640 10 monclient: build_initial_monmap 2024-05-15T13:20:55.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.487+0000 7f03abdfe640 10 monclient: monmap: 2024-05-15T13:20:55.486 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T13:20:55.487 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:55.487 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T13:20:55.488715+0000 2024-05-15T13:20:55.487 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T13:20:55.488715+0000 2024-05-15T13:20:55.487 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T13:20:55.487 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:55.487 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T13:20:55.487 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T13:20:55.487 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T13:20:55.487 INFO:teuthology.orchestra.run.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T13:20:55.487 INFO:teuthology.orchestra.run.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T13:20:55.487 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:55.488 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.488+0000 7f03abdfe640 1 Processor -- start 2024-05-15T13:20:55.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.488+0000 7f03abdfe640 1 -- start start 2024-05-15T13:20:55.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.488+0000 7f03abdfe640 10 monclient: init 2024-05-15T13:20:55.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.489+0000 7f03abdfe640 10 monclient: _reopen_session rank -1 2024-05-15T13:20:55.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.489+0000 7f03abdfe640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-05-15T13:20:55.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.489+0000 7f03abdfe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f03a406bd70 addr v1:172.21.15.117:6789/0 2024-05-15T13:20:55.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.489+0000 7f03abdfe640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f03a4070b80 0x7f03a4142940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:55.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.489+0000 7f03abdfe640 10 monclient(hunting): picked mon.noname-b con 0x7f03a4070b80 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T13:20:55.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.489+0000 7f03abdfe640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f03a40ad820 0x7f03a40adcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:55.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.489+0000 7f03abdfe640 10 monclient(hunting): picked mon.noname-a con 0x7f03a40ad820 addr v2:172.21.15.117:3300/0 2024-05-15T13:20:55.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.489+0000 7f03abdfe640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03a40ae1f0 con 0x7f03a406bd70 2024-05-15T13:20:55.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.489+0000 7f03abdfe640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:55.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.489+0000 7f03abdfe640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03a406c150 con 0x7f03a40ad820 2024-05-15T13:20:55.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.489+0000 7f03abdfe640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:55.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.489+0000 7f03abdfe640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03a406fab0 con 0x7f03a4070b80 2024-05-15T13:20:55.490 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.489+0000 7f03abdfe640 10 monclient(hunting): _renew_subs 2024-05-15T13:20:55.490 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.489+0000 7f03abdfe640 10 monclient(hunting): authenticate will time out at 3618.730938s 2024-05-15T13:20:55.490 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.489+0000 7f03aa39b640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f03a40ad820 0x7f03a40adcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:55.490 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.490+0000 7f03a9b9a640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f03a406bd70 0x7f03a4072570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:44294/0 (socket says 172.21.15.117:44294) 2024-05-15T13:20:55.490 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.490+0000 7f03a9b9a640 1 -- 172.21.15.117:0/3615433802 learned_addr learned my addr 172.21.15.117:0/3615433802 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-05-15T13:20:55.490 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.490+0000 7f03aa39b640 10 monclient(hunting): get_auth_request con 0x7f03a40ad820 auth_method 0 2024-05-15T13:20:55.490 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.490+0000 7f03aa39b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:55.490 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.490+0000 7f03aa39b640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:55.490 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.490+0000 7f03aa39b640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:55.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.491+0000 7f03aa39b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:55.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.491+0000 7f03aa39b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:55.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.491+0000 7f03aa39b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:55.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.494+0000 7f03aa39b640 10 monclient(hunting): handle_auth_done global_id 16110 payload 274 2024-05-15T13:20:55.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.494+0000 7f03aa39b640 1 -- 172.21.15.117:0/3615433802 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f03a4070b80 msgr2=0x7f03a4142940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T13:20:55.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.494+0000 7f03aa39b640 1 --2- 172.21.15.117:0/3615433802 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f03a4070b80 0x7f03a4142940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:55.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.494+0000 7f03aa39b640 1 -- 172.21.15.117:0/3615433802 >> v1:172.21.15.117:6789/0 conn(0x7f03a406bd70 legacy=0x7f03a4072570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:55.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.494+0000 7f03aa39b640 10 monclient: _finish_hunting 0 2024-05-15T13:20:55.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.494+0000 7f03aa39b640 1 monclient: found mon.noname-a 2024-05-15T13:20:55.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.494+0000 7f03aa39b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:20:55.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.494+0000 7f03aa39b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.117:3300/0 2024-05-15T13:20:55.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.494+0000 7f03aa39b640 1 -- 172.21.15.117:0/3615433802 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03a4072c80 con 0x7f03a40ad820 2024-05-15T13:20:55.496 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.497+0000 7f03aa39b640 1 --2- 172.21.15.117:0/3615433802 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f03a40ad820 0x7f03a40adcb0 secure :-1 s=READY pgs=3312 cs=0 l=1 rev1=1 crypto rx=0x7f039c009dc0 tx=0x7f039c02be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be075757afd4178d server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:55.498 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.499+0000 7f03a8b98640 1 -- 172.21.15.117:0/3615433802 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f039c039070 con 0x7f03a40ad820 2024-05-15T13:20:55.498 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.499+0000 7f03a8b98640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:55.498 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.499+0000 7f03a8b98640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T13:20:55.498 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.499+0000 7f03a8b98640 10 monclient: dump: 2024-05-15T13:20:55.499 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:55.499 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:55.499 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:55.499 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:55.499 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:55.499 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:55.499 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:55.499 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:55.499 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:55.499 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:55.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.499+0000 7f03a8b98640 10 monclient: _finish_auth 0 2024-05-15T13:20:55.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.499+0000 7f03a8b98640 10 monclient: _check_auth_tickets 2024-05-15T13:20:55.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.499+0000 7f03a8b98640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:20:55.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.499+0000 7f03abdfe640 5 monclient: authenticate success, global_id 16110 2024-05-15T13:20:55.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.499+0000 7f03abdfe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T13:20:55.501 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.501+0000 7f03a8b98640 1 -- 172.21.15.117:0/3615433802 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f039c02cc40 con 0x7f03a40ad820 2024-05-15T13:20:55.501 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.501+0000 7f03a8b98640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:20:55.501 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.501+0000 7f03a8b98640 1 -- 172.21.15.117:0/3615433802 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f039c034440 con 0x7f03a40ad820 2024-05-15T13:20:55.501 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.501+0000 7f03abdfe640 10 monclient: get_monmap_and_config success 2024-05-15T13:20:55.501 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.501+0000 7f03abdfe640 4 set_mon_vals no callback set 2024-05-15T13:20:55.501 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.501+0000 7f03a8b98640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:55.501 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.501+0000 7f03a8b98640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T13:20:55.501 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.501+0000 7f03a8b98640 10 monclient: dump: 2024-05-15T13:20:55.501 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:55.501 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:55.501 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:55.501 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:55.501 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:55.502 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:55.502 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:55.502 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:55.502 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:55.502 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:55.502 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.501+0000 7f03abdfe640 10 monclient: shutdown 2024-05-15T13:20:55.502 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.501+0000 7f03abdfe640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:20:55.502 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.501+0000 7f03abdfe640 1 -- 172.21.15.117:0/3615433802 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f03a40ad820 msgr2=0x7f03a40adcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:55.502 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.501+0000 7f03abdfe640 1 --2- 172.21.15.117:0/3615433802 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f03a40ad820 0x7f03a40adcb0 secure :-1 s=READY pgs=3312 cs=0 l=1 rev1=1 crypto rx=0x7f039c009dc0 tx=0x7f039c02be90 comp rx=0 tx=0).stop 2024-05-15T13:20:55.502 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.501+0000 7f03abdfe640 1 -- 172.21.15.117:0/3615433802 shutdown_connections 2024-05-15T13:20:55.502 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.501+0000 7f03abdfe640 1 --2- 172.21.15.117:0/3615433802 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f03a40ad820 0x7f03a40adcb0 unknown :-1 s=CLOSED pgs=3312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:55.502 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.501+0000 7f03abdfe640 1 --2- 172.21.15.117:0/3615433802 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f03a4070b80 0x7f03a4142940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:55.502 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.501+0000 7f03abdfe640 1 -- 172.21.15.117:0/3615433802 >> 172.21.15.117:0/3615433802 conn(0x7f03a406ada0 msgr2=0x7f03a406b1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:20:55.502 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.502+0000 7f03abdfe640 1 -- 172.21.15.117:0/3615433802 shutdown_connections 2024-05-15T13:20:55.503 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.502+0000 7f03abdfe640 1 -- 172.21.15.117:0/3615433802 wait complete. 2024-05-15T13:20:55.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.502+0000 7f03abdfe640 10 monclient: build_initial_monmap 2024-05-15T13:20:55.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.502+0000 7f03abdfe640 10 monclient: monmap: 2024-05-15T13:20:55.505 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T13:20:55.505 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T13:20:55.505 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-05-15T13:20:55.505 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-05-15T13:20:55.505 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T13:20:55.505 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:55.505 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T13:20:55.505 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T13:20:55.505 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T13:20:55.505 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:55.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.503+0000 7f03abdfe640 1 librados: starting msgr at 2024-05-15T13:20:55.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.503+0000 7f03abdfe640 1 librados: starting objecter 2024-05-15T13:20:55.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.503+0000 7f03abdfe640 1 Processor -- start 2024-05-15T13:20:55.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.503+0000 7f03abdfe640 1 -- start start 2024-05-15T13:20:55.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.503+0000 7f03abdfe640 1 librados: setting wanted keys 2024-05-15T13:20:55.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.503+0000 7f03abdfe640 1 librados: calling monclient init 2024-05-15T13:20:55.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.503+0000 7f03abdfe640 10 monclient: init 2024-05-15T13:20:55.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.503+0000 7f03abdfe640 10 monclient: _reopen_session rank -1 2024-05-15T13:20:55.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.503+0000 7f03abdfe640 10 monclient: _add_conns ranks=[0,2,1] 2024-05-15T13:20:55.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.503+0000 7f03abdfe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f03a406bd70 0x7f03a4136820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:55.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.503+0000 7f03abdfe640 10 monclient(hunting): picked mon.noname-a con 0x7f03a406bd70 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T13:20:55.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.503+0000 7f03abdfe640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f03a40ad820 0x7f03a41db890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:55.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.503+0000 7f03abdfe640 10 monclient(hunting): picked mon.noname-c con 0x7f03a40ad820 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T13:20:55.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.503+0000 7f03abdfe640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f03a41dbdd0 0x7f03a41d81b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:55.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.503+0000 7f03abdfe640 10 monclient(hunting): picked mon.noname-b con 0x7f03a41dbdd0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T13:20:55.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.503+0000 7f03abdfe640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:55.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.503+0000 7f03abdfe640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03a406f6b0 con 0x7f03a406bd70 2024-05-15T13:20:55.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.503+0000 7f03abdfe640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:55.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.503+0000 7f03abdfe640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03a413cf40 con 0x7f03a41dbdd0 2024-05-15T13:20:55.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.503+0000 7f03abdfe640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:55.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.503+0000 7f03abdfe640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03a4149cd0 con 0x7f03a40ad820 2024-05-15T13:20:55.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.503+0000 7f03abdfe640 10 monclient(hunting): _renew_subs 2024-05-15T13:20:55.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.503+0000 7f03abdfe640 10 monclient(hunting): authenticate will time out at 3618.745787s 2024-05-15T13:20:55.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.504+0000 7f03a9b9a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f03a406bd70 0x7f03a4136820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:55.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.504+0000 7f03a9b9a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f03a406bd70 0x7f03a4136820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:38852/0 (socket says 172.21.15.117:38852) 2024-05-15T13:20:55.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.504+0000 7f03a9b9a640 1 -- 172.21.15.117:0/3367701042 learned_addr learned my addr 172.21.15.117:0/3367701042 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T13:20:55.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.504+0000 7f03a9b9a640 10 monclient(hunting): get_auth_request con 0x7f03a406bd70 auth_method 0 2024-05-15T13:20:55.508 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.504+0000 7f03a9b9a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:55.508 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.504+0000 7f03a9b9a640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:55.508 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.504+0000 7f03a9b9a640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:55.508 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.505+0000 7f03a9b9a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:55.508 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.505+0000 7f03a9b9a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:55.508 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.505+0000 7f03a9b9a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:55.509 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.508+0000 7f03a9b9a640 10 monclient(hunting): handle_auth_done global_id 16116 payload 824 2024-05-15T13:20:55.509 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.508+0000 7f03a9b9a640 1 -- 172.21.15.117:0/3367701042 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f03a40ad820 msgr2=0x7f03a41db890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:55.509 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.508+0000 7f03a9b9a640 1 --2- 172.21.15.117:0/3367701042 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f03a40ad820 0x7f03a41db890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:55.509 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.508+0000 7f03a9b9a640 1 -- 172.21.15.117:0/3367701042 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f03a41dbdd0 msgr2=0x7f03a41d81b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:55.509 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.508+0000 7f03a9b9a640 1 --2- 172.21.15.117:0/3367701042 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f03a41dbdd0 0x7f03a41d81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:55.509 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.508+0000 7f03a9b9a640 10 monclient: _finish_hunting 0 2024-05-15T13:20:55.509 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.508+0000 7f03a9b9a640 1 monclient: found mon.noname-a 2024-05-15T13:20:55.509 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.508+0000 7f03a9b9a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:20:55.510 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.508+0000 7f03a9b9a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.117:3300/0 2024-05-15T13:20:55.510 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.508+0000 7f03a9b9a640 1 -- 172.21.15.117:0/3367701042 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03a41dc280 con 0x7f03a406bd70 2024-05-15T13:20:55.510 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.510+0000 7f03a9b9a640 1 --2- 172.21.15.117:0/3367701042 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f03a406bd70 0x7f03a4136820 secure :-1 s=READY pgs=3313 cs=0 l=1 rev1=1 crypto rx=0x7f03a000c400 tx=0x7f03a000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:55.510 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.510+0000 7f039a7fc640 10 client.?.objecter ms_handle_connect 0x7f03a406bd70 2024-05-15T13:20:55.510 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.510+0000 7f039a7fc640 10 client.?.objecter resend_mon_ops 2024-05-15T13:20:55.513 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.513+0000 7f039a7fc640 1 -- 172.21.15.117:0/3367701042 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a0017070 con 0x7f03a406bd70 2024-05-15T13:20:55.513 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.513+0000 7f039a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:55.513 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.513+0000 7f039a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T13:20:55.513 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.513+0000 7f039a7fc640 10 monclient: dump: 2024-05-15T13:20:55.513 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:55.513 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:55.513 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:55.513 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:55.513 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:55.513 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:55.513 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:55.513 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:55.513 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:55.513 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:55.514 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.513+0000 7f039a7fc640 10 monclient: _finish_auth 0 2024-05-15T13:20:55.514 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.513+0000 7f039a7fc640 10 monclient: _check_auth_tickets 2024-05-15T13:20:55.514 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.513+0000 7f039a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:20:55.514 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.513+0000 7f039a7fc640 1 -- 172.21.15.117:0/3367701042 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03a0005ce0 con 0x7f03a406bd70 2024-05-15T13:20:55.514 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.513+0000 7f039a7fc640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:20:55.514 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.513+0000 7f039a7fc640 1 -- 172.21.15.117:0/3367701042 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a00072f0 con 0x7f03a406bd70 2024-05-15T13:20:55.514 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.513+0000 7f039a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:55.514 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.513+0000 7f039a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T13:20:55.514 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.513+0000 7f039a7fc640 10 monclient: dump: 2024-05-15T13:20:55.514 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:55.514 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:55.514 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:55.515 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:55.515 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:55.515 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:55.515 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:55.515 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:55.515 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:55.515 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:55.515 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.513+0000 7f03abdfe640 5 monclient: authenticate success, global_id 16116 2024-05-15T13:20:55.515 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.513+0000 7f03abdfe640 10 monclient: _renew_subs 2024-05-15T13:20:55.515 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.513+0000 7f03abdfe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T13:20:55.515 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.513+0000 7f03abdfe640 1 -- 172.21.15.117:0/3367701042 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03a413e160 con 0x7f03a406bd70 2024-05-15T13:20:55.515 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.513+0000 7f039bfff640 4 set_mon_vals no callback set 2024-05-15T13:20:55.515 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.513+0000 7f03abdfe640 10 client.16116.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:20:55.516 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.513+0000 7f03abdfe640 10 monclient: _renew_subs 2024-05-15T13:20:55.516 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.513+0000 7f03abdfe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T13:20:55.516 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.513+0000 7f03abdfe640 1 -- 172.21.15.117:0/3367701042 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03a41d8a20 con 0x7f03a406bd70 2024-05-15T13:20:55.516 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.513+0000 7f03abdfe640 1 librados: init done 2024-05-15T13:20:55.516 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.514+0000 7f03abdfe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T13:20:55.516 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.514+0000 7f03abdfe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:20:55.516 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.514+0000 7f03abdfe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T13:20:55.516 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.514+0000 7f03abdfe640 1 -- 172.21.15.117:0/3367701042 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03a404f0e0 con 0x7f03a406bd70 2024-05-15T13:20:55.761 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.762+0000 7f039a7fc640 1 -- 172.21.15.117:0/3367701042 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f03a001a070 con 0x7f03a406bd70 2024-05-15T13:20:55.761 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.762+0000 7f039a7fc640 1 --2- 172.21.15.117:0/3367701042 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f038806ceb0 0x7f038806f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:55.761 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.762+0000 7f03a9399640 1 --2- 172.21.15.117:0/3367701042 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f038806ceb0 0x7f038806f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:55.761 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.762+0000 7f03a9399640 10 monclient: get_auth_request con 0x7f038806ceb0 auth_method 0 2024-05-15T13:20:55.762 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.763+0000 7f03a9399640 1 --2- 172.21.15.117:0/3367701042 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f038806ceb0 0x7f038806f370 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f03a413d3e0 tx=0x7f0394006c90 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:55.762 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.763+0000 7f039a7fc640 10 client.16116.objecter ms_handle_connect 0x7f038806ceb0 2024-05-15T13:20:55.778 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.779+0000 7f039a7fc640 1 -- 172.21.15.117:0/3367701042 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(1225..1225 src has 912..1225) v4 ==== 6365+0+0 (secure 0 0 0) 0x7f03a0056290 con 0x7f03a406bd70 2024-05-15T13:20:55.778 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.779+0000 7f039a7fc640 10 client.16116.objecter ms_dispatch 0x7f03a4000b90 osd_map(1225..1225 src has 912..1225) v4 2024-05-15T13:20:55.778 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.779+0000 7f039a7fc640 3 client.16116.objecter handle_osd_map got epochs [1225,1225] > 0 2024-05-15T13:20:55.778 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.779+0000 7f039a7fc640 3 client.16116.objecter handle_osd_map decoding full epoch 1225 2024-05-15T13:20:55.778 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.779+0000 7f039a7fc640 20 client.16116.objecter dump_active .. 0 homeless 2024-05-15T13:20:55.778 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.779+0000 7f039a7fc640 1 -- 172.21.15.117:0/3367701042 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f03a0021020 con 0x7f03a406bd70 2024-05-15T13:20:55.778 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.779+0000 7f039a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:20:55.778 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.779+0000 7f039a7fc640 10 monclient: _finish_command 1 = system:0 2024-05-15T13:20:55.947 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.948+0000 7f03abdfe640 1 -- 172.21.15.117:0/3367701042 --> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f03a41d7f10 con 0x7f038806ceb0 2024-05-15T13:20:55.950 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.951+0000 7f039a7fc640 1 -- 172.21.15.117:0/3367701042 <== mgr.4107 v2:172.21.15.117:6800/1785766523 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+116159 (secure 0 0 0) 0x7f03a41d7f10 con 0x7f038806ceb0 2024-05-15T13:20:55.951 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-05-15T13:20:55.952 INFO:teuthology.orchestra.run.smithi117.stderr:dumped all 2024-05-15T13:20:55.954 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.953+0000 7f03abdfe640 10 librados: watch_flush enter 2024-05-15T13:20:55.954 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.954+0000 7f03abdfe640 10 librados: watch_flush exit 2024-05-15T13:20:55.954 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.954+0000 7f03abdfe640 20 client.16116.objecter shutdown clearing up homeless session... 2024-05-15T13:20:55.954 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.954+0000 7f03abdfe640 10 client.16116.objecter successfully canceled tick 2024-05-15T13:20:55.954 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.954+0000 7f03abdfe640 1 -- 172.21.15.117:0/3367701042 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f038806ceb0 msgr2=0x7f038806f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:55.954 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.954+0000 7f03abdfe640 1 --2- 172.21.15.117:0/3367701042 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f038806ceb0 0x7f038806f370 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f03a413d3e0 tx=0x7f0394006c90 comp rx=0 tx=0).stop 2024-05-15T13:20:55.955 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.954+0000 7f03abdfe640 10 monclient: shutdown 2024-05-15T13:20:55.955 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.954+0000 7f03abdfe640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:20:55.955 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.954+0000 7f03abdfe640 1 -- 172.21.15.117:0/3367701042 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f03a406bd70 msgr2=0x7f03a4136820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:55.955 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.954+0000 7f03abdfe640 1 --2- 172.21.15.117:0/3367701042 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f03a406bd70 0x7f03a4136820 secure :-1 s=READY pgs=3313 cs=0 l=1 rev1=1 crypto rx=0x7f03a000c400 tx=0x7f03a000c8c0 comp rx=0 tx=0).stop 2024-05-15T13:20:55.955 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.954+0000 7f03abdfe640 1 -- 172.21.15.117:0/3367701042 shutdown_connections 2024-05-15T13:20:55.955 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.954+0000 7f03abdfe640 1 --2- 172.21.15.117:0/3367701042 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f038806ceb0 0x7f038806f370 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:55.955 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.954+0000 7f03abdfe640 1 --2- 172.21.15.117:0/3367701042 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f03a41dbdd0 0x7f03a41d81b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:55.955 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.954+0000 7f03abdfe640 1 --2- 172.21.15.117:0/3367701042 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f03a40ad820 0x7f03a41db890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:55.955 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.954+0000 7f03abdfe640 1 --2- 172.21.15.117:0/3367701042 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f03a406bd70 0x7f03a4136820 unknown :-1 s=CLOSED pgs=3313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:55.955 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.954+0000 7f03abdfe640 1 -- 172.21.15.117:0/3367701042 >> 172.21.15.117:0/3367701042 conn(0x7f03a406ada0 msgr2=0x7f03a4138780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:20:55.955 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.954+0000 7f03abdfe640 1 -- 172.21.15.117:0/3367701042 shutdown_connections 2024-05-15T13:20:55.955 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.954+0000 7f03abdfe640 1 -- 172.21.15.117:0/3367701042 wait complete. 2024-05-15T13:20:55.955 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:55.954+0000 7f03abdfe640 1 librados: shutdown 2024-05-15T13:20:55.964 INFO:teuthology.orchestra.run.smithi117.stdout:{"pg_ready":true,"pg_map":{"version":2371,"stamp":"2024-05-15T13:20:54.619156+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":602520,"kb_used_data":42200,"kb_used_omap":12,"kb_used_meta":560243,"kb_avail":754372200,"statfs":{"total":773094113280,"available":772477132800,"internally_reserved":0,"allocated":43212800,"data_stored":21573962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":573689394},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002598"},"pg_stats":[{"pgid":"6.16","version":"0'0","reported_seq":2373,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:36.377820+0000","last_change":"2024-05-15T13:20:36.378566+0000","last_active":"2024-05-15T13:20:36.377820+0000","last_peered":"2024-05-15T13:20:36.377820+0000","last_clean":"2024-05-15T13:20:36.377820+0000","last_became_active":"2024-05-15T12:44:59.550282+0000","last_became_peered":"2024-05-15T12:44:59.550282+0000","last_unstale":"2024-05-15T13:20:36.377820+0000","last_undegraded":"2024-05-15T13:20:36.377820+0000","last_fullsized":"2024-05-15T13:20:36.377820+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:20:36.376385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:20:36.376385+0000","last_clean_scrub_stamp":"2024-05-15T13:20:36.376385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:42:02.630914+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":2388,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:30.954548+0000","last_change":"2024-05-15T13:20:30.955358+0000","last_active":"2024-05-15T13:20:30.954548+0000","last_peered":"2024-05-15T13:20:30.954548+0000","last_clean":"2024-05-15T13:20:30.954548+0000","last_became_active":"2024-05-15T12:44:51.202412+0000","last_became_peered":"2024-05-15T12:44:51.202412+0000","last_unstale":"2024-05-15T13:20:30.954548+0000","last_undegraded":"2024-05-15T13:20:30.954548+0000","last_fullsized":"2024-05-15T13:20:30.954548+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:20:30.952959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:20:30.952959+0000","last_clean_scrub_stamp":"2024-05-15T13:20:30.952959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T16:35:53.838886+0000","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":2374,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:38.372672+0000","last_change":"2024-05-15T13:20:38.373417+0000","last_active":"2024-05-15T13:20:38.372672+0000","last_peered":"2024-05-15T13:20:38.372672+0000","last_clean":"2024-05-15T13:20:38.372672+0000","last_became_active":"2024-05-15T12:45:11.468881+0000","last_became_peered":"2024-05-15T12:45:11.468881+0000","last_unstale":"2024-05-15T13:20:38.372672+0000","last_undegraded":"2024-05-15T13:20:38.372672+0000","last_fullsized":"2024-05-15T13:20:38.372672+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:20:38.371142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:20:38.371142+0000","last_clean_scrub_stamp":"2024-05-15T13:20:38.371142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:45:24.885798+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":2375,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:30.604998+0000","last_change":"2024-05-15T13:20:30.605827+0000","last_active":"2024-05-15T13:20:30.604998+0000","last_peered":"2024-05-15T13:20:30.604998+0000","last_clean":"2024-05-15T13:20:30.604998+0000","last_became_active":"2024-05-15T12:44:55.775514+0000","last_became_peered":"2024-05-15T12:44:55.775514+0000","last_unstale":"2024-05-15T13:20:30.604998+0000","last_undegraded":"2024-05-15T13:20:30.604998+0000","last_fullsized":"2024-05-15T13:20:30.604998+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:20:30.603365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:20:30.603365+0000","last_clean_scrub_stamp":"2024-05-15T13:20:30.603365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:22:04.910315+0000","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":2373,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:30.440339+0000","last_change":"2024-05-15T13:20:30.441124+0000","last_active":"2024-05-15T13:20:30.440339+0000","last_peered":"2024-05-15T13:20:30.440339+0000","last_clean":"2024-05-15T13:20:30.440339+0000","last_became_active":"2024-05-15T12:44:57.673614+0000","last_became_peered":"2024-05-15T12:44:57.673614+0000","last_unstale":"2024-05-15T13:20:30.440339+0000","last_undegraded":"2024-05-15T13:20:30.440339+0000","last_fullsized":"2024-05-15T13:20:30.440339+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:20:30.438699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:20:30.438699+0000","last_clean_scrub_stamp":"2024-05-15T13:20:30.438699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T15:26:33.578850+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":2375,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:35.386758+0000","last_change":"2024-05-15T13:20:35.386758+0000","last_active":"2024-05-15T13:20:35.386758+0000","last_peered":"2024-05-15T13:20:35.386758+0000","last_clean":"2024-05-15T13:20:35.386758+0000","last_became_active":"2024-05-15T12:44:59.480451+0000","last_became_peered":"2024-05-15T12:44:59.480451+0000","last_unstale":"2024-05-15T13:20:35.386758+0000","last_undegraded":"2024-05-15T13:20:35.386758+0000","last_fullsized":"2024-05-15T13:20:35.386758+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:20:35.384485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:20:35.384485+0000","last_clean_scrub_stamp":"2024-05-15T13:20:35.384485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-17T00:43:26.076197+0000","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":2378,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:37.988321+0000","last_change":"2024-05-15T13:20:37.989101+0000","last_active":"2024-05-15T13:20:37.988321+0000","last_peered":"2024-05-15T13:20:37.988321+0000","last_clean":"2024-05-15T13:20:37.988321+0000","last_became_active":"2024-05-15T12:44:54.914910+0000","last_became_peered":"2024-05-15T12:44:54.914910+0000","last_unstale":"2024-05-15T13:20:37.988321+0000","last_undegraded":"2024-05-15T13:20:37.988321+0000","last_fullsized":"2024-05-15T13:20:37.988321+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:20:37.986829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:20:37.986829+0000","last_clean_scrub_stamp":"2024-05-15T13:20:37.986829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:00:21.030741+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":2375,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:37.397316+0000","last_change":"2024-05-15T13:20:37.398100+0000","last_active":"2024-05-15T13:20:37.397316+0000","last_peered":"2024-05-15T13:20:37.397316+0000","last_clean":"2024-05-15T13:20:37.397316+0000","last_became_active":"2024-05-15T12:44:59.595062+0000","last_became_peered":"2024-05-15T12:44:59.595062+0000","last_unstale":"2024-05-15T13:20:37.397316+0000","last_undegraded":"2024-05-15T13:20:37.397316+0000","last_fullsized":"2024-05-15T13:20:37.397316+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:20:37.395668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:20:37.395668+0000","last_clean_scrub_stamp":"2024-05-15T13:20:37.395668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-17T01:06:08.755358+0000","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":2390,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:31.904837+0000","last_change":"2024-05-15T13:20:31.905645+0000","last_active":"2024-05-15T13:20:31.904837+0000","last_peered":"2024-05-15T13:20:31.904837+0000","last_clean":"2024-05-15T13:20:31.904837+0000","last_became_active":"2024-05-15T12:44:51.785998+0000","last_became_peered":"2024-05-15T12:44:51.785998+0000","last_unstale":"2024-05-15T13:20:31.904837+0000","last_undegraded":"2024-05-15T13:20:31.904837+0000","last_fullsized":"2024-05-15T13:20:31.904837+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:20:31.901241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:20:31.901241+0000","last_clean_scrub_stamp":"2024-05-15T13:20:31.901241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T23:38:13.971102+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":2388,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:35.912850+0000","last_change":"2024-05-15T13:20:35.913616+0000","last_active":"2024-05-15T13:20:35.912850+0000","last_peered":"2024-05-15T13:20:35.912850+0000","last_clean":"2024-05-15T13:20:35.912850+0000","last_became_active":"2024-05-15T12:44:53.608407+0000","last_became_peered":"2024-05-15T12:44:53.608407+0000","last_unstale":"2024-05-15T13:20:35.912850+0000","last_undegraded":"2024-05-15T13:20:35.912850+0000","last_fullsized":"2024-05-15T13:20:35.912850+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:20:35.911330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:20:35.911330+0000","last_clean_scrub_stamp":"2024-05-15T13:20:35.911330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:49:17.960383+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":2376,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:35.180786+0000","last_change":"2024-05-15T13:20:35.181575+0000","last_active":"2024-05-15T13:20:35.180786+0000","last_peered":"2024-05-15T13:20:35.180786+0000","last_clean":"2024-05-15T13:20:35.180786+0000","last_became_active":"2024-05-15T12:44:52.212480+0000","last_became_peered":"2024-05-15T12:44:52.212480+0000","last_unstale":"2024-05-15T13:20:35.180786+0000","last_undegraded":"2024-05-15T13:20:35.180786+0000","last_fullsized":"2024-05-15T13:20:35.180786+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:20:35.179218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:20:35.179218+0000","last_clean_scrub_stamp":"2024-05-15T13:20:35.179218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:41:27.561770+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":2390,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:32.925935+0000","last_change":"2024-05-15T13:20:32.926731+0000","last_active":"2024-05-15T13:20:32.925935+0000","last_peered":"2024-05-15T13:20:32.925935+0000","last_clean":"2024-05-15T13:20:32.925935+0000","last_became_active":"2024-05-15T12:44:56.197837+0000","last_became_peered":"2024-05-15T12:44:56.197837+0000","last_unstale":"2024-05-15T13:20:32.925935+0000","last_undegraded":"2024-05-15T13:20:32.925935+0000","last_fullsized":"2024-05-15T13:20:32.925935+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:20:32.924348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:20:32.924348+0000","last_clean_scrub_stamp":"2024-05-15T13:20:32.924348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T23:44:19.826872+0000","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":2377,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:37.062308+0000","last_change":"2024-05-15T13:20:37.063109+0000","last_active":"2024-05-15T13:20:37.062308+0000","last_peered":"2024-05-15T13:20:37.062308+0000","last_clean":"2024-05-15T13:20:37.062308+0000","last_became_active":"2024-05-15T12:44:53.377773+0000","last_became_peered":"2024-05-15T12:44:53.377773+0000","last_unstale":"2024-05-15T13:20:37.062308+0000","last_undegraded":"2024-05-15T13:20:37.062308+0000","last_fullsized":"2024-05-15T13:20:37.062308+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:20:37.060606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:20:37.060606+0000","last_clean_scrub_stamp":"2024-05-15T13:20:37.060606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:22:59.331764+0000","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"1225'78","reported_seq":2414,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:29.968940+0000","last_change":"2024-05-15T13:20:29.970241+0000","last_active":"2024-05-15T13:20:29.968940+0000","last_peered":"2024-05-15T13:20:29.968940+0000","last_clean":"2024-05-15T13:20:29.968940+0000","last_became_active":"2024-05-15T12:43:04.148735+0000","last_became_peered":"2024-05-15T12:43:04.148735+0000","last_unstale":"2024-05-15T13:20:29.968940+0000","last_undegraded":"2024-05-15T13:20:29.968940+0000","last_fullsized":"2024-05-15T13:20:29.968940+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"1225'78","last_scrub_stamp":"2024-05-15T13:20:29.967300+0000","last_deep_scrub":"1225'78","last_deep_scrub_stamp":"2024-05-15T13:20:29.967300+0000","last_clean_scrub_stamp":"2024-05-15T13:20:29.967300+0000","objects_scrubbed":2,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T16:05:45.963490+0000","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2388,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:36.027919+0000","last_change":"2024-05-15T13:20:36.027919+0000","last_active":"2024-05-15T13:20:36.027919+0000","last_peered":"2024-05-15T13:20:36.027919+0000","last_clean":"2024-05-15T13:20:36.027919+0000","last_became_active":"2024-05-15T12:43:09.445878+0000","last_became_peered":"2024-05-15T12:43:09.445878+0000","last_unstale":"2024-05-15T13:20:36.027919+0000","last_undegraded":"2024-05-15T13:20:36.027919+0000","last_fullsized":"2024-05-15T13:20:36.027919+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:20:36.025653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:20:36.025653+0000","last_clean_scrub_stamp":"2024-05-15T13:20:36.025653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:33:45.196358+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":2383,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:36.147639+0000","last_change":"2024-05-15T13:20:36.148442+0000","last_active":"2024-05-15T13:20:36.147639+0000","last_peered":"2024-05-15T13:20:36.147639+0000","last_clean":"2024-05-15T13:20:36.147639+0000","last_became_active":"2024-05-15T12:44:50.696664+0000","last_became_peered":"2024-05-15T12:44:50.696664+0000","last_unstale":"2024-05-15T13:20:36.147639+0000","last_undegraded":"2024-05-15T13:20:36.147639+0000","last_fullsized":"2024-05-15T13:20:36.147639+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:20:36.146035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:20:36.146035+0000","last_clean_scrub_stamp":"2024-05-15T13:20:36.146035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T16:33:43.983650+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2388,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:32.048223+0000","last_change":"2024-05-15T13:20:32.049290+0000","last_active":"2024-05-15T13:20:32.048223+0000","last_peered":"2024-05-15T13:20:32.048223+0000","last_clean":"2024-05-15T13:20:32.048223+0000","last_became_active":"2024-05-15T12:43:08.677751+0000","last_became_peered":"2024-05-15T12:43:08.677751+0000","last_unstale":"2024-05-15T13:20:32.048223+0000","last_undegraded":"2024-05-15T13:20:32.048223+0000","last_fullsized":"2024-05-15T13:20:32.048223+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:20:32.046530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:20:32.046530+0000","last_clean_scrub_stamp":"2024-05-15T13:20:32.046530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:15:02.487152+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":2374,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:29.500929+0000","last_change":"2024-05-15T13:20:29.500929+0000","last_active":"2024-05-15T13:20:29.500929+0000","last_peered":"2024-05-15T13:20:29.500929+0000","last_clean":"2024-05-15T13:20:29.500929+0000","last_became_active":"2024-05-15T12:44:57.552111+0000","last_became_peered":"2024-05-15T12:44:57.552111+0000","last_unstale":"2024-05-15T13:20:29.500929+0000","last_undegraded":"2024-05-15T13:20:29.500929+0000","last_fullsized":"2024-05-15T13:20:29.500929+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:20:29.495828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:20:29.495828+0000","last_clean_scrub_stamp":"2024-05-15T13:20:29.495828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:57:37.144480+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2392,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:35.118696+0000","last_change":"2024-05-15T13:20:35.118696+0000","last_active":"2024-05-15T13:20:35.118696+0000","last_peered":"2024-05-15T13:20:35.118696+0000","last_clean":"2024-05-15T13:20:35.118696+0000","last_became_active":"2024-05-15T12:43:08.600745+0000","last_became_peered":"2024-05-15T12:43:08.600745+0000","last_unstale":"2024-05-15T13:20:35.118696+0000","last_undegraded":"2024-05-15T13:20:35.118696+0000","last_fullsized":"2024-05-15T13:20:35.118696+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:20:35.114816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:20:35.114816+0000","last_clean_scrub_stamp":"2024-05-15T13:20:35.114816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:11:27.397223+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":2389,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:34.919067+0000","last_change":"2024-05-15T13:20:34.919842+0000","last_active":"2024-05-15T13:20:34.919067+0000","last_peered":"2024-05-15T13:20:34.919067+0000","last_clean":"2024-05-15T13:20:34.919067+0000","last_became_active":"2024-05-15T12:44:51.354879+0000","last_became_peered":"2024-05-15T12:44:51.354879+0000","last_unstale":"2024-05-15T13:20:34.919067+0000","last_undegraded":"2024-05-15T13:20:34.919067+0000","last_fullsized":"2024-05-15T13:20:34.919067+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:20:34.917445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:20:34.917445+0000","last_clean_scrub_stamp":"2024-05-15T13:20:34.917445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T15:53:27.835991+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2388,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:34.060369+0000","last_change":"2024-05-15T13:20:34.060369+0000","last_active":"2024-05-15T13:20:34.060369+0000","last_peered":"2024-05-15T13:20:34.060369+0000","last_clean":"2024-05-15T13:20:34.060369+0000","last_became_active":"2024-05-15T12:43:09.363876+0000","last_became_peered":"2024-05-15T12:43:09.363876+0000","last_unstale":"2024-05-15T13:20:34.060369+0000","last_undegraded":"2024-05-15T13:20:34.060369+0000","last_fullsized":"2024-05-15T13:20:34.060369+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:20:34.057618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:20:34.057618+0000","last_clean_scrub_stamp":"2024-05-15T13:20:34.057618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:13:32.574075+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":2383,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:38.107074+0000","last_change":"2024-05-15T13:20:38.107854+0000","last_active":"2024-05-15T13:20:38.107074+0000","last_peered":"2024-05-15T13:20:38.107074+0000","last_clean":"2024-05-15T13:20:38.107074+0000","last_became_active":"2024-05-15T12:44:54.179460+0000","last_became_peered":"2024-05-15T12:44:54.179460+0000","last_unstale":"2024-05-15T13:20:38.107074+0000","last_undegraded":"2024-05-15T13:20:38.107074+0000","last_fullsized":"2024-05-15T13:20:38.107074+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:20:38.105464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:20:38.105464+0000","last_clean_scrub_stamp":"2024-05-15T13:20:38.105464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:45:25.736404+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2399,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:33.879142+0000","last_change":"2024-05-15T13:20:33.879142+0000","last_active":"2024-05-15T13:20:33.879142+0000","last_peered":"2024-05-15T13:20:33.879142+0000","last_clean":"2024-05-15T13:20:33.879142+0000","last_became_active":"2024-05-15T12:43:09.377725+0000","last_became_peered":"2024-05-15T12:43:09.377725+0000","last_unstale":"2024-05-15T13:20:33.879142+0000","last_undegraded":"2024-05-15T13:20:33.879142+0000","last_fullsized":"2024-05-15T13:20:33.879142+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:20:33.876427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:20:33.876427+0000","last_clean_scrub_stamp":"2024-05-15T13:20:33.876427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:56:33.255420+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"1225'8","reported_seq":2392,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:39.328877+0000","last_change":"2024-05-15T13:20:39.330128+0000","last_active":"2024-05-15T13:20:39.328877+0000","last_peered":"2024-05-15T13:20:39.328877+0000","last_clean":"2024-05-15T13:20:39.328877+0000","last_became_active":"2024-05-15T12:44:58.318306+0000","last_became_peered":"2024-05-15T12:44:58.318306+0000","last_unstale":"2024-05-15T13:20:39.328877+0000","last_undegraded":"2024-05-15T13:20:39.328877+0000","last_fullsized":"2024-05-15T13:20:39.328877+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"1225'8","last_scrub_stamp":"2024-05-15T13:20:39.327335+0000","last_deep_scrub":"1225'8","last_deep_scrub_stamp":"2024-05-15T13:20:39.327335+0000","last_clean_scrub_stamp":"2024-05-15T13:20:39.327335+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:31:22.179540+0000","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2387,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:33.037347+0000","last_change":"2024-05-15T13:20:33.038128+0000","last_active":"2024-05-15T13:20:33.037347+0000","last_peered":"2024-05-15T13:20:33.037347+0000","last_clean":"2024-05-15T13:20:33.037347+0000","last_became_active":"2024-05-15T12:43:08.773614+0000","last_became_peered":"2024-05-15T12:43:08.773614+0000","last_unstale":"2024-05-15T13:20:33.037347+0000","last_undegraded":"2024-05-15T13:20:33.037347+0000","last_fullsized":"2024-05-15T13:20:33.037347+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:20:33.035675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:20:33.035675+0000","last_clean_scrub_stamp":"2024-05-15T13:20:33.035675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:11:27.596306+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":2379,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:41.060005+0000","last_change":"2024-05-15T13:20:41.060808+0000","last_active":"2024-05-15T13:20:41.060005+0000","last_peered":"2024-05-15T13:20:41.060005+0000","last_clean":"2024-05-15T13:20:41.060005+0000","last_became_active":"2024-05-15T12:45:08.661140+0000","last_became_peered":"2024-05-15T12:45:08.661140+0000","last_unstale":"2024-05-15T13:20:41.060005+0000","last_undegraded":"2024-05-15T13:20:41.060005+0000","last_fullsized":"2024-05-15T13:20:41.060005+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:20:41.055860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:20:41.055860+0000","last_clean_scrub_stamp":"2024-05-15T13:20:41.055860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T23:33:29.175455+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2385,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:32.323304+0000","last_change":"2024-05-15T13:20:32.324086+0000","last_active":"2024-05-15T13:20:32.323304+0000","last_peered":"2024-05-15T13:20:32.323304+0000","last_clean":"2024-05-15T13:20:32.323304+0000","last_became_active":"2024-05-15T12:43:09.448237+0000","last_became_peered":"2024-05-15T12:43:09.448237+0000","last_unstale":"2024-05-15T13:20:32.323304+0000","last_undegraded":"2024-05-15T13:20:32.323304+0000","last_fullsized":"2024-05-15T13:20:32.323304+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:20:32.321603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:20:32.321603+0000","last_clean_scrub_stamp":"2024-05-15T13:20:32.321603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:59:20.645448+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":2374,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:33.572743+0000","last_change":"2024-05-15T13:20:33.572743+0000","last_active":"2024-05-15T13:20:33.572743+0000","last_peered":"2024-05-15T13:20:33.572743+0000","last_clean":"2024-05-15T13:20:33.572743+0000","last_became_active":"2024-05-15T12:45:18.642568+0000","last_became_peered":"2024-05-15T12:45:18.642568+0000","last_unstale":"2024-05-15T13:20:33.572743+0000","last_undegraded":"2024-05-15T13:20:33.572743+0000","last_fullsized":"2024-05-15T13:20:33.572743+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:20:33.570424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:20:33.570424+0000","last_clean_scrub_stamp":"2024-05-15T13:20:33.570424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:42:13.307627+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2386,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:35.035346+0000","last_change":"2024-05-15T13:20:35.036109+0000","last_active":"2024-05-15T13:20:35.035346+0000","last_peered":"2024-05-15T13:20:35.035346+0000","last_clean":"2024-05-15T13:20:35.035346+0000","last_became_active":"2024-05-15T12:43:09.424460+0000","last_became_peered":"2024-05-15T12:43:09.424460+0000","last_unstale":"2024-05-15T13:20:35.035346+0000","last_undegraded":"2024-05-15T13:20:35.035346+0000","last_fullsized":"2024-05-15T13:20:35.035346+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:20:35.033770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:20:35.033770+0000","last_clean_scrub_stamp":"2024-05-15T13:20:35.033770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:33:18.468857+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":2376,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:34.166398+0000","last_change":"2024-05-15T13:20:34.167162+0000","last_active":"2024-05-15T13:20:34.166398+0000","last_peered":"2024-05-15T13:20:34.166398+0000","last_clean":"2024-05-15T13:20:34.166398+0000","last_became_active":"2024-05-15T12:44:52.088745+0000","last_became_peered":"2024-05-15T12:44:52.088745+0000","last_unstale":"2024-05-15T13:20:34.166398+0000","last_undegraded":"2024-05-15T13:20:34.166398+0000","last_fullsized":"2024-05-15T13:20:34.166398+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:20:34.163575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:20:34.163575+0000","last_clean_scrub_stamp":"2024-05-15T13:20:34.163575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:51:21.080373+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":2374,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:33.349317+0000","last_change":"2024-05-15T13:20:33.350103+0000","last_active":"2024-05-15T13:20:33.349317+0000","last_peered":"2024-05-15T13:20:33.349317+0000","last_clean":"2024-05-15T13:20:33.349317+0000","last_became_active":"2024-05-15T12:44:56.470246+0000","last_became_peered":"2024-05-15T12:44:56.470246+0000","last_unstale":"2024-05-15T13:20:33.349317+0000","last_undegraded":"2024-05-15T13:20:33.349317+0000","last_fullsized":"2024-05-15T13:20:33.349317+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:20:33.345686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:20:33.345686+0000","last_clean_scrub_stamp":"2024-05-15T13:20:33.345686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:17:14.984898+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":2380,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:37.141716+0000","last_change":"2024-05-15T13:20:37.142571+0000","last_active":"2024-05-15T13:20:37.141716+0000","last_peered":"2024-05-15T13:20:37.141716+0000","last_clean":"2024-05-15T13:20:37.141716+0000","last_became_active":"2024-05-15T12:44:51.660527+0000","last_became_peered":"2024-05-15T12:44:51.660527+0000","last_unstale":"2024-05-15T13:20:37.141716+0000","last_undegraded":"2024-05-15T13:20:37.141716+0000","last_fullsized":"2024-05-15T13:20:37.141716+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:20:37.140042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:20:37.140042+0000","last_clean_scrub_stamp":"2024-05-15T13:20:37.140042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:39:00.087259+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":2372,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:31.448097+0000","last_change":"2024-05-15T13:20:31.448896+0000","last_active":"2024-05-15T13:20:31.448097+0000","last_peered":"2024-05-15T13:20:31.448097+0000","last_clean":"2024-05-15T13:20:31.448097+0000","last_became_active":"2024-05-15T12:45:11.998256+0000","last_became_peered":"2024-05-15T12:45:11.998256+0000","last_unstale":"2024-05-15T13:20:31.448097+0000","last_undegraded":"2024-05-15T13:20:31.448097+0000","last_fullsized":"2024-05-15T13:20:31.448097+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:20:31.446430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:20:31.446430+0000","last_clean_scrub_stamp":"2024-05-15T13:20:31.446430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:19:55.009137+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":2374,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:37.297143+0000","last_change":"2024-05-15T13:20:37.298439+0000","last_active":"2024-05-15T13:20:37.297143+0000","last_peered":"2024-05-15T13:20:37.297143+0000","last_clean":"2024-05-15T13:20:37.297143+0000","last_became_active":"2024-05-15T12:44:55.557214+0000","last_became_peered":"2024-05-15T12:44:55.557214+0000","last_unstale":"2024-05-15T13:20:37.297143+0000","last_undegraded":"2024-05-15T13:20:37.297143+0000","last_fullsized":"2024-05-15T13:20:37.297143+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:20:37.294611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:20:37.294611+0000","last_clean_scrub_stamp":"2024-05-15T13:20:37.294611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:00:34.714418+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":2376,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:38.979534+0000","last_change":"2024-05-15T13:20:38.980340+0000","last_active":"2024-05-15T13:20:38.979534+0000","last_peered":"2024-05-15T13:20:38.979534+0000","last_clean":"2024-05-15T13:20:38.979534+0000","last_became_active":"2024-05-15T12:44:56.072618+0000","last_became_peered":"2024-05-15T12:44:56.072618+0000","last_unstale":"2024-05-15T13:20:38.979534+0000","last_undegraded":"2024-05-15T13:20:38.979534+0000","last_fullsized":"2024-05-15T13:20:38.979534+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:20:38.977912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:20:38.977912+0000","last_clean_scrub_stamp":"2024-05-15T13:20:38.977912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:29:18.706711+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":2378,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:38.277422+0000","last_change":"2024-05-15T13:20:38.278180+0000","last_active":"2024-05-15T13:20:38.277422+0000","last_peered":"2024-05-15T13:20:38.277422+0000","last_clean":"2024-05-15T13:20:38.277422+0000","last_became_active":"2024-05-15T12:44:55.739811+0000","last_became_peered":"2024-05-15T12:44:55.739811+0000","last_unstale":"2024-05-15T13:20:38.277422+0000","last_undegraded":"2024-05-15T13:20:38.277422+0000","last_fullsized":"2024-05-15T13:20:38.277422+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:20:38.275523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:20:38.275523+0000","last_clean_scrub_stamp":"2024-05-15T13:20:38.275523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:18:34.807593+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":2378,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:40.030499+0000","last_change":"2024-05-15T13:20:40.031857+0000","last_active":"2024-05-15T13:20:40.030499+0000","last_peered":"2024-05-15T13:20:40.030499+0000","last_clean":"2024-05-15T13:20:40.030499+0000","last_became_active":"2024-05-15T12:44:56.429996+0000","last_became_peered":"2024-05-15T12:44:56.429996+0000","last_unstale":"2024-05-15T13:20:40.030499+0000","last_undegraded":"2024-05-15T13:20:40.030499+0000","last_fullsized":"2024-05-15T13:20:40.030499+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:20:40.028044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:20:40.028044+0000","last_clean_scrub_stamp":"2024-05-15T13:20:40.028044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:56:26.728102+0000","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":2371,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:32.524702+0000","last_change":"2024-05-15T13:20:32.525514+0000","last_active":"2024-05-15T13:20:32.524702+0000","last_peered":"2024-05-15T13:20:32.524702+0000","last_clean":"2024-05-15T13:20:32.524702+0000","last_became_active":"2024-05-15T12:45:18.546529+0000","last_became_peered":"2024-05-15T12:45:18.546529+0000","last_unstale":"2024-05-15T13:20:32.524702+0000","last_undegraded":"2024-05-15T13:20:32.524702+0000","last_fullsized":"2024-05-15T13:20:32.524702+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:20:32.523103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:20:32.523103+0000","last_clean_scrub_stamp":"2024-05-15T13:20:32.523103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T15:20:37.568315+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":2373,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:34.321957+0000","last_change":"2024-05-15T13:20:34.322744+0000","last_active":"2024-05-15T13:20:34.321957+0000","last_peered":"2024-05-15T13:20:34.321957+0000","last_clean":"2024-05-15T13:20:34.321957+0000","last_became_active":"2024-05-15T12:44:56.634257+0000","last_became_peered":"2024-05-15T12:44:56.634257+0000","last_unstale":"2024-05-15T13:20:34.321957+0000","last_undegraded":"2024-05-15T13:20:34.321957+0000","last_fullsized":"2024-05-15T13:20:34.321957+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:20:34.320366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:20:34.320366+0000","last_clean_scrub_stamp":"2024-05-15T13:20:34.320366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:05:34.279395+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":2375,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:36.272623+0000","last_change":"2024-05-15T13:20:36.273437+0000","last_active":"2024-05-15T13:20:36.272623+0000","last_peered":"2024-05-15T13:20:36.272623+0000","last_clean":"2024-05-15T13:20:36.272623+0000","last_became_active":"2024-05-15T12:44:52.483168+0000","last_became_peered":"2024-05-15T12:44:52.483168+0000","last_unstale":"2024-05-15T13:20:36.272623+0000","last_undegraded":"2024-05-15T13:20:36.272623+0000","last_fullsized":"2024-05-15T13:20:36.272623+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:20:36.269350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:20:36.269350+0000","last_clean_scrub_stamp":"2024-05-15T13:20:36.269350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T23:37:11.059432+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":2376,"reported_epoch":1225,"state":"active+clean","last_fresh":"2024-05-15T13:20:42.014517+0000","last_change":"2024-05-15T13:20:42.015338+0000","last_active":"2024-05-15T13:20:42.014517+0000","last_peered":"2024-05-15T13:20:42.014517+0000","last_clean":"2024-05-15T13:20:42.014517+0000","last_became_active":"2024-05-15T12:45:12.190202+0000","last_became_peered":"2024-05-15T12:45:12.190202+0000","last_unstale":"2024-05-15T13:20:42.014517+0000","last_undegraded":"2024-05-15T13:20:42.014517+0000","last_fullsized":"2024-05-15T13:20:42.014517+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:20:42.009192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:20:42.009192+0000","last_clean_scrub_stamp":"2024-05-15T13:20:42.009192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:11:26.492311+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":20,"seq":85899346379,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72840,"kb_used_data":5568,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299000,"statfs":{"total":96636764160,"available":96562176000,"internally_reserved":0,"allocated":5701632,"data_stored":2995124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":20,"seq":85899346378,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68288,"kb_used_data":5176,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303552,"statfs":{"total":96636764160,"available":96566837248,"internally_reserved":0,"allocated":5300224,"data_stored":2597284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":20,"seq":85899346378,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94592,"kb_used_data":5176,"kb_used_omap":1,"kb_used_meta":89406,"kb_avail":94277248,"statfs":{"total":96636764160,"available":96539901952,"internally_reserved":0,"allocated":5300224,"data_stored":2597284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":91552198},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":19,"seq":81604379082,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72452,"kb_used_data":5180,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299388,"statfs":{"total":96636764160,"available":96562573312,"internally_reserved":0,"allocated":5304320,"data_stored":2597289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":19,"seq":81604379082,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72448,"kb_used_data":5176,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299392,"statfs":{"total":96636764160,"available":96562577408,"internally_reserved":0,"allocated":5300224,"data_stored":2597284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509896,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72452,"kb_used_data":5180,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299388,"statfs":{"total":96636764160,"available":96562573312,"internally_reserved":0,"allocated":5304320,"data_stored":2597289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542601,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72840,"kb_used_data":5568,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299000,"statfs":{"total":96636764160,"available":96562176000,"internally_reserved":0,"allocated":5701632,"data_stored":2995124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":13,"seq":55834575306,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76608,"kb_used_data":5176,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94295232,"statfs":{"total":96636764160,"available":96558317568,"internally_reserved":0,"allocated":5300224,"data_stored":2597284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T13:20:55.967 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-05-15T13:20:56.071 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.072+0000 7f0c9b1c6640 10 monclient: get_monmap_and_config 2024-05-15T13:20:56.071 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.072+0000 7f0c9b1c6640 10 monclient: build_initial_monmap 2024-05-15T13:20:56.071 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.072+0000 7f0c9b1c6640 10 monclient: monmap: 2024-05-15T13:20:56.071 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T13:20:56.071 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:56.071 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T13:20:56.073385+0000 2024-05-15T13:20:56.071 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T13:20:56.073385+0000 2024-05-15T13:20:56.071 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T13:20:56.072 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:56.072 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a-legacy 2024-05-15T13:20:56.072 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2024-05-15T13:20:56.072 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-b 2024-05-15T13:20:56.072 INFO:teuthology.orchestra.run.smithi117.stderr:3: v2:172.21.15.117:3300/0 mon.noname-a 2024-05-15T13:20:56.072 INFO:teuthology.orchestra.run.smithi117.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2024-05-15T13:20:56.072 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:56.074 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.073+0000 7f0c9b1c6640 1 Processor -- start 2024-05-15T13:20:56.074 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.073+0000 7f0c9b1c6640 1 -- start start 2024-05-15T13:20:56.074 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.073+0000 7f0c9b1c6640 10 monclient: init 2024-05-15T13:20:56.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.073+0000 7f0c9b1c6640 10 monclient: _reopen_session rank -1 2024-05-15T13:20:56.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.073+0000 7f0c9b1c6640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-05-15T13:20:56.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.073+0000 7f0c9b1c6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0c9406c900 addr v1:172.21.15.117:6789/0 2024-05-15T13:20:56.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.073+0000 7f0c9b1c6640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f0c940af1c0 0x7f0c94142c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:56.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.073+0000 7f0c9b1c6640 10 monclient(hunting): picked mon.noname-a con 0x7f0c940af1c0 addr v2:172.21.15.117:3300/0 2024-05-15T13:20:56.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.073+0000 7f0c9b1c6640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f0c941431b0 0x7f0c94143620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:56.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.073+0000 7f0c9b1c6640 10 monclient(hunting): picked mon.noname-c con 0x7f0c941431b0 addr v2:172.21.15.178:3300/0 2024-05-15T13:20:56.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.073+0000 7f0c9b1c6640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c940ae0d0 con 0x7f0c9406c900 2024-05-15T13:20:56.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.073+0000 7f0c9b1c6640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:56.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.073+0000 7f0c9b1c6640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c9406cce0 con 0x7f0c940af1c0 2024-05-15T13:20:56.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.073+0000 7f0c9b1c6640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:56.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.073+0000 7f0c9b1c6640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c940b0500 con 0x7f0c941431b0 2024-05-15T13:20:56.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.073+0000 7f0c9b1c6640 10 monclient(hunting): _renew_subs 2024-05-15T13:20:56.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.073+0000 7f0c9b1c6640 10 monclient(hunting): authenticate will time out at 3619.315646s 2024-05-15T13:20:56.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.074+0000 7f0c93fff640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f0c940af1c0 0x7f0c94142c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:56.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.074+0000 7f0c99763640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f0c941431b0 0x7f0c94143620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:56.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.074+0000 7f0c93fff640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f0c940af1c0 0x7f0c94142c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:38858/0 (socket says 172.21.15.117:38858) 2024-05-15T13:20:56.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.074+0000 7f0c93fff640 1 -- 172.21.15.117:0/3920220117 learned_addr learned my addr 172.21.15.117:0/3920220117 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T13:20:56.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.074+0000 7f0c93fff640 10 monclient(hunting): get_auth_request con 0x7f0c940af1c0 auth_method 0 2024-05-15T13:20:56.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.074+0000 7f0c93fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:56.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.074+0000 7f0c93fff640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:56.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.074+0000 7f0c93fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:56.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.074+0000 7f0c99763640 10 monclient(hunting): get_auth_request con 0x7f0c941431b0 auth_method 0 2024-05-15T13:20:56.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.074+0000 7f0c99763640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:56.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.074+0000 7f0c99763640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:56.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.074+0000 7f0c99763640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:56.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.075+0000 7f0c93fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:56.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.075+0000 7f0c93fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:56.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.075+0000 7f0c93fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:56.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.075+0000 7f0c99763640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:56.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.075+0000 7f0c99763640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:56.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.075+0000 7f0c99763640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:56.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.077+0000 7f0c93fff640 10 monclient(hunting): handle_auth_done global_id 16119 payload 274 2024-05-15T13:20:56.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.077+0000 7f0c93fff640 1 -- 172.21.15.117:0/3920220117 >> v2:172.21.15.178:3300/0 conn(0x7f0c941431b0 msgr2=0x7f0c94143620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T13:20:56.078 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.077+0000 7f0c93fff640 1 --2- 172.21.15.117:0/3920220117 >> v2:172.21.15.178:3300/0 conn(0x7f0c941431b0 0x7f0c94143620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:56.078 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.077+0000 7f0c93fff640 1 -- 172.21.15.117:0/3920220117 >> v1:172.21.15.117:6789/0 conn(0x7f0c9406c900 legacy=0x7f0c940aeab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:56.078 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.077+0000 7f0c93fff640 10 monclient: _finish_hunting 0 2024-05-15T13:20:56.078 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.077+0000 7f0c93fff640 1 monclient: found mon.noname-a 2024-05-15T13:20:56.078 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.077+0000 7f0c93fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:20:56.078 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.077+0000 7f0c93fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.117:3300/0 2024-05-15T13:20:56.078 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.077+0000 7f0c93fff640 1 -- 172.21.15.117:0/3920220117 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c940ae3b0 con 0x7f0c940af1c0 2024-05-15T13:20:56.080 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.082+0000 7f0c93fff640 1 --2- 172.21.15.117:0/3920220117 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c940af1c0 0x7f0c94142c70 secure :-1 s=READY pgs=3315 cs=0 l=1 rev1=1 crypto rx=0x7f0c940ad970 tx=0x7f0c8802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bff0d6e3b9979bfb server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:56.084 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.085+0000 7f0c937fe640 1 -- 172.21.15.117:0/3920220117 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c8802c9f0 con 0x7f0c940af1c0 2024-05-15T13:20:56.085 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.085+0000 7f0c937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:56.085 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.085+0000 7f0c937fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T13:20:56.085 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.085+0000 7f0c937fe640 10 monclient: dump: 2024-05-15T13:20:56.085 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:56.085 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:56.085 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:56.085 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:56.085 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:56.085 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:56.085 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:56.085 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:56.085 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:56.085 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:56.086 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.085+0000 7f0c937fe640 10 monclient: _finish_auth 0 2024-05-15T13:20:56.086 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.085+0000 7f0c937fe640 10 monclient: _check_auth_tickets 2024-05-15T13:20:56.086 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.085+0000 7f0c937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:20:56.086 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.085+0000 7f0c937fe640 1 -- 172.21.15.117:0/3920220117 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c8802cb90 con 0x7f0c940af1c0 2024-05-15T13:20:56.086 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.085+0000 7f0c9b1c6640 5 monclient: authenticate success, global_id 16119 2024-05-15T13:20:56.086 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.085+0000 7f0c9b1c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T13:20:56.086 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.085+0000 7f0c937fe640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:20:56.086 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.085+0000 7f0c9b1c6640 10 monclient: get_monmap_and_config success 2024-05-15T13:20:56.086 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.085+0000 7f0c9b1c6640 4 set_mon_vals no callback set 2024-05-15T13:20:56.086 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.085+0000 7f0c937fe640 1 -- 172.21.15.117:0/3920220117 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c880343f0 con 0x7f0c940af1c0 2024-05-15T13:20:56.086 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.085+0000 7f0c937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:56.086 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.085+0000 7f0c937fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T13:20:56.086 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.085+0000 7f0c937fe640 10 monclient: dump: 2024-05-15T13:20:56.086 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:56.087 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:56.087 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:56.087 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:56.087 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:56.087 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:56.087 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:56.087 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:56.087 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:56.087 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:56.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.086+0000 7f0c9b1c6640 10 monclient: shutdown 2024-05-15T13:20:56.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.086+0000 7f0c9b1c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:20:56.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.086+0000 7f0c9b1c6640 1 -- 172.21.15.117:0/3920220117 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c940af1c0 msgr2=0x7f0c94142c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:56.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.086+0000 7f0c9b1c6640 1 --2- 172.21.15.117:0/3920220117 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c940af1c0 0x7f0c94142c70 secure :-1 s=READY pgs=3315 cs=0 l=1 rev1=1 crypto rx=0x7f0c940ad970 tx=0x7f0c8802be50 comp rx=0 tx=0).stop 2024-05-15T13:20:56.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.086+0000 7f0c9b1c6640 1 -- 172.21.15.117:0/3920220117 shutdown_connections 2024-05-15T13:20:56.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.086+0000 7f0c9b1c6640 1 --2- 172.21.15.117:0/3920220117 >> v2:172.21.15.178:3300/0 conn(0x7f0c941431b0 0x7f0c94143620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:56.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.086+0000 7f0c9b1c6640 1 --2- 172.21.15.117:0/3920220117 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c940af1c0 0x7f0c94142c70 unknown :-1 s=CLOSED pgs=3315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:56.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.086+0000 7f0c9b1c6640 1 -- 172.21.15.117:0/3920220117 >> 172.21.15.117:0/3920220117 conn(0x7f0c940b2ac0 msgr2=0x7f0c940b2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:20:56.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.086+0000 7f0c9b1c6640 1 -- 172.21.15.117:0/3920220117 shutdown_connections 2024-05-15T13:20:56.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.086+0000 7f0c9b1c6640 1 -- 172.21.15.117:0/3920220117 wait complete. 2024-05-15T13:20:56.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.087+0000 7f0c9b1c6640 10 monclient: build_initial_monmap 2024-05-15T13:20:56.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.087+0000 7f0c9b1c6640 10 monclient: monmap: 2024-05-15T13:20:56.091 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-05-15T13:20:56.091 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T13:20:56.091 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-05-15T13:20:56.091 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-05-15T13:20:56.091 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-05-15T13:20:56.091 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:56.091 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-a 2024-05-15T13:20:56.091 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2024-05-15T13:20:56.091 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-c 2024-05-15T13:20:56.091 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:56.091 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.087+0000 7f0c9b1c6640 1 librados: starting msgr at 2024-05-15T13:20:56.091 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.087+0000 7f0c9b1c6640 1 librados: starting objecter 2024-05-15T13:20:56.091 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.087+0000 7f0c9b1c6640 1 Processor -- start 2024-05-15T13:20:56.091 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.087+0000 7f0c9b1c6640 1 -- start start 2024-05-15T13:20:56.091 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.087+0000 7f0c9b1c6640 1 librados: setting wanted keys 2024-05-15T13:20:56.091 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.087+0000 7f0c9b1c6640 1 librados: calling monclient init 2024-05-15T13:20:56.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.087+0000 7f0c9b1c6640 10 monclient: init 2024-05-15T13:20:56.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.088+0000 7f0c9b1c6640 10 monclient: _reopen_session rank -1 2024-05-15T13:20:56.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.088+0000 7f0c9b1c6640 10 monclient: _add_conns ranks=[0,1,2] 2024-05-15T13:20:56.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.088+0000 7f0c9b1c6640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c9406c900 0x7f0c94145f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:56.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.088+0000 7f0c9b1c6640 10 monclient(hunting): picked mon.noname-a con 0x7f0c9406c900 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-05-15T13:20:56.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.088+0000 7f0c9b1c6640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0c940af1c0 0x7f0c9411f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:56.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.088+0000 7f0c9b1c6640 10 monclient(hunting): picked mon.noname-b con 0x7f0c940af1c0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-05-15T13:20:56.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.088+0000 7f0c9b1c6640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0c941431b0 0x7f0c941d7840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:56.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.088+0000 7f0c9b1c6640 10 monclient(hunting): picked mon.noname-c con 0x7f0c941431b0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-05-15T13:20:56.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.088+0000 7f0c9b1c6640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:56.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.088+0000 7f0c9b1c6640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c940af610 con 0x7f0c9406c900 2024-05-15T13:20:56.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.088+0000 7f0c9b1c6640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:56.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.088+0000 7f0c9b1c6640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c9406d730 con 0x7f0c940af1c0 2024-05-15T13:20:56.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.088+0000 7f0c9b1c6640 10 monclient(hunting): start opening mon connection 2024-05-15T13:20:56.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.088+0000 7f0c9b1c6640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c940af790 con 0x7f0c941431b0 2024-05-15T13:20:56.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.088+0000 7f0c9b1c6640 10 monclient(hunting): _renew_subs 2024-05-15T13:20:56.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.088+0000 7f0c9b1c6640 10 monclient(hunting): authenticate will time out at 3619.330475s 2024-05-15T13:20:56.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.089+0000 7f0c99763640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0c941431b0 0x7f0c941d7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:56.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.089+0000 7f0c98f62640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c9406c900 0x7f0c94145f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:56.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.089+0000 7f0c93fff640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0c940af1c0 0x7f0c9411f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:56.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.089+0000 7f0c98f62640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c9406c900 0x7f0c94145f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:38868/0 (socket says 172.21.15.117:38868) 2024-05-15T13:20:56.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.089+0000 7f0c99763640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0c941431b0 0x7f0c941d7840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.117:36638/0 (socket says 172.21.15.117:36638) 2024-05-15T13:20:56.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.089+0000 7f0c98f62640 1 -- 172.21.15.117:0/3544704919 learned_addr learned my addr 172.21.15.117:0/3544704919 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-05-15T13:20:56.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.089+0000 7f0c98f62640 10 monclient(hunting): get_auth_request con 0x7f0c9406c900 auth_method 0 2024-05-15T13:20:56.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.089+0000 7f0c93fff640 1 --2- 172.21.15.117:0/3544704919 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0c940af1c0 0x7f0c9411f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:41180/0 (socket says 172.21.15.117:41180) 2024-05-15T13:20:56.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.089+0000 7f0c98f62640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:56.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.089+0000 7f0c98f62640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:56.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.089+0000 7f0c98f62640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:56.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.089+0000 7f0c99763640 10 monclient(hunting): get_auth_request con 0x7f0c941431b0 auth_method 0 2024-05-15T13:20:56.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.089+0000 7f0c99763640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:56.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.089+0000 7f0c99763640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:56.094 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.089+0000 7f0c99763640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:56.094 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.089+0000 7f0c93fff640 10 monclient(hunting): get_auth_request con 0x7f0c940af1c0 auth_method 0 2024-05-15T13:20:56.094 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.089+0000 7f0c93fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:20:56.094 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.089+0000 7f0c93fff640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:20:56.094 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.089+0000 7f0c93fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:20:56.094 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.090+0000 7f0c98f62640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:56.094 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.090+0000 7f0c98f62640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:56.094 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.090+0000 7f0c98f62640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:56.094 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.090+0000 7f0c99763640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:56.094 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.090+0000 7f0c99763640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:56.094 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.090+0000 7f0c99763640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:56.094 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.090+0000 7f0c93fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:20:56.094 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.090+0000 7f0c93fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:20:56.094 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.090+0000 7f0c93fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:20:56.095 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.093+0000 7f0c98f62640 10 monclient(hunting): handle_auth_done global_id 16125 payload 824 2024-05-15T13:20:56.095 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.093+0000 7f0c98f62640 1 -- 172.21.15.117:0/3544704919 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0c941431b0 msgr2=0x7f0c941d7840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:56.095 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.093+0000 7f0c98f62640 1 --2- 172.21.15.117:0/3544704919 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0c941431b0 0x7f0c941d7840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:56.096 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.093+0000 7f0c98f62640 1 -- 172.21.15.117:0/3544704919 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0c940af1c0 msgr2=0x7f0c9411f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:56.096 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.093+0000 7f0c98f62640 1 --2- 172.21.15.117:0/3544704919 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0c940af1c0 0x7f0c9411f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:56.096 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.093+0000 7f0c98f62640 10 monclient: _finish_hunting 0 2024-05-15T13:20:56.096 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.093+0000 7f0c98f62640 1 monclient: found mon.noname-a 2024-05-15T13:20:56.096 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.093+0000 7f0c98f62640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:20:56.096 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.093+0000 7f0c98f62640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.117:3300/0 2024-05-15T13:20:56.096 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.093+0000 7f0c98f62640 1 -- 172.21.15.117:0/3544704919 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c9413efb0 con 0x7f0c9406c900 2024-05-15T13:20:56.096 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.095+0000 7f0c98f62640 1 --2- 172.21.15.117:0/3544704919 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c9406c900 0x7f0c94145f50 secure :-1 s=READY pgs=3316 cs=0 l=1 rev1=1 crypto rx=0x7f0c8400c400 tx=0x7f0c8400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:56.096 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.095+0000 7f0c917fa640 10 client.?.objecter ms_handle_connect 0x7f0c9406c900 2024-05-15T13:20:56.096 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.095+0000 7f0c917fa640 10 client.?.objecter resend_mon_ops 2024-05-15T13:20:56.099 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.099+0000 7f0c917fa640 1 -- 172.21.15.117:0/3544704919 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c840052d0 con 0x7f0c9406c900 2024-05-15T13:20:56.099 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.099+0000 7f0c917fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:56.099 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.099+0000 7f0c917fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T13:20:56.099 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.099+0000 7f0c917fa640 10 monclient: dump: 2024-05-15T13:20:56.099 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:56.099 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:56.099 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:56.099 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:56.099 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:56.099 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:56.099 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:56.100 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:56.100 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:56.100 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:56.100 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.099+0000 7f0c917fa640 10 monclient: _finish_auth 0 2024-05-15T13:20:56.100 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.099+0000 7f0c917fa640 10 monclient: _check_auth_tickets 2024-05-15T13:20:56.100 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.099+0000 7f0c917fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:20:56.100 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.099+0000 7f0c917fa640 1 -- 172.21.15.117:0/3544704919 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c84005490 con 0x7f0c9406c900 2024-05-15T13:20:56.100 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.099+0000 7f0c9b1c6640 5 monclient: authenticate success, global_id 16125 2024-05-15T13:20:56.100 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.099+0000 7f0c9b1c6640 10 monclient: _renew_subs 2024-05-15T13:20:56.100 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.099+0000 7f0c9b1c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T13:20:56.100 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.099+0000 7f0c9b1c6640 1 -- 172.21.15.117:0/3544704919 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c9411f1a0 con 0x7f0c9406c900 2024-05-15T13:20:56.100 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.099+0000 7f0c917fa640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:20:56.100 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.099+0000 7f0c917fa640 1 -- 172.21.15.117:0/3544704919 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c84007260 con 0x7f0c9406c900 2024-05-15T13:20:56.100 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.099+0000 7f0c917fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:20:56.100 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.099+0000 7f0c917fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T13:20:56.101 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.099+0000 7f0c917fa640 10 monclient: dump: 2024-05-15T13:20:56.101 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-05-15T13:20:56.101 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 8817f6cf-d933-4825-b972-575ef59f5ba9 2024-05-15T13:20:56.101 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:56.101 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-05-15T12:41:21.922077+0000 2024-05-15T13:20:56.101 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-05-15T13:20:56.101 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-05-15T13:20:56.101 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-05-15T13:20:56.101 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-05-15T13:20:56.101 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2024-05-15T13:20:56.101 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:20:56.101 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.099+0000 7f0c9b1c6640 10 client.16125.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:20:56.101 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.099+0000 7f0c92ffd640 4 set_mon_vals no callback set 2024-05-15T13:20:56.101 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.099+0000 7f0c9b1c6640 10 monclient: _renew_subs 2024-05-15T13:20:56.101 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.099+0000 7f0c9b1c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T13:20:56.101 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.099+0000 7f0c9b1c6640 1 -- 172.21.15.117:0/3544704919 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c941d7d80 con 0x7f0c9406c900 2024-05-15T13:20:56.102 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.099+0000 7f0c9b1c6640 1 librados: init done 2024-05-15T13:20:56.102 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.100+0000 7f0c9b1c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T13:20:56.102 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.100+0000 7f0c9b1c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:20:56.102 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.100+0000 7f0c9b1c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T13:20:56.102 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.100+0000 7f0c9b1c6640 1 -- 172.21.15.117:0/3544704919 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c9404f0e0 con 0x7f0c9406c900 2024-05-15T13:20:56.347 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.347+0000 7f0c917fa640 1 -- 172.21.15.117:0/3544704919 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f0c84007400 con 0x7f0c9406c900 2024-05-15T13:20:56.347 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.348+0000 7f0c917fa640 1 --2- 172.21.15.117:0/3544704919 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f0c8006ce90 0x7f0c8006f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:20:56.347 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.348+0000 7f0c93fff640 1 --2- 172.21.15.117:0/3544704919 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f0c8006ce90 0x7f0c8006f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:20:56.347 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.348+0000 7f0c93fff640 10 monclient: get_auth_request con 0x7f0c8006ce90 auth_method 0 2024-05-15T13:20:56.347 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.348+0000 7f0c917fa640 10 client.16125.objecter ms_handle_connect 0x7f0c8006ce90 2024-05-15T13:20:56.347 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.348+0000 7f0c93fff640 1 --2- 172.21.15.117:0/3544704919 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f0c8006ce90 0x7f0c8006f350 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f0c88035000 tx=0x7f0c88030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:20:56.373 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.374+0000 7f0c917fa640 1 -- 172.21.15.117:0/3544704919 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(1225..1225 src has 912..1225) v4 ==== 6365+0+0 (secure 0 0 0) 0x7f0c840564a0 con 0x7f0c9406c900 2024-05-15T13:20:56.374 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.374+0000 7f0c917fa640 10 client.16125.objecter ms_dispatch 0x7f0c94000b90 osd_map(1225..1225 src has 912..1225) v4 2024-05-15T13:20:56.374 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.374+0000 7f0c917fa640 3 client.16125.objecter handle_osd_map got epochs [1225,1225] > 0 2024-05-15T13:20:56.374 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.374+0000 7f0c917fa640 3 client.16125.objecter handle_osd_map decoding full epoch 1225 2024-05-15T13:20:56.374 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.374+0000 7f0c917fa640 20 client.16125.objecter dump_active .. 0 homeless 2024-05-15T13:20:56.374 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.374+0000 7f0c917fa640 1 -- 172.21.15.117:0/3544704919 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f0c84058400 con 0x7f0c9406c900 2024-05-15T13:20:56.374 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.374+0000 7f0c917fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:20:56.375 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.374+0000 7f0c917fa640 10 monclient: _finish_command 1 = system:0 2024-05-15T13:20:56.557 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.558+0000 7f0c9b1c6640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-05-15T13:20:56.557 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.558+0000 7f0c9b1c6640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-05-15T13:20:56.557 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.558+0000 7f0c9b1c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.117:3300/0 2024-05-15T13:20:56.557 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.558+0000 7f0c9b1c6640 1 -- 172.21.15.117:0/3544704919 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f0c94120150 con 0x7f0c9406c900 2024-05-15T13:20:56.648 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.649+0000 7f0c917fa640 1 -- 172.21.15.117:0/3544704919 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v7)=0 v7) v1 ==== 123+0+0 (secure 0 0 0) 0x7f0c8405c4a0 con 0x7f0c9406c900 2024-05-15T13:20:56.648 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.649+0000 7f0c917fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-05-15T13:20:56.648 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.649+0000 7f0c917fa640 10 monclient: _finish_command 2 = system:0 2024-05-15T13:20:56.649 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.649+0000 7f0c917fa640 1 -- 172.21.15.117:0/3544704919 <== mon.0 v2:172.21.15.117:3300/0 8 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f0c84060a60 con 0x7f0c9406c900 2024-05-15T13:20:56.649 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.649+0000 7f0c917fa640 10 monclient: handle_config config(1 keys) v1 2024-05-15T13:20:56.649 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.649+0000 7f0c927fc640 4 set_mon_vals no callback set 2024-05-15T13:20:56.649 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.649+0000 7f0c927fc640 10 set_mon_vals mon_health_to_clog = false 2024-05-15T13:20:56.651 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.651+0000 7f0c9b1c6640 10 librados: watch_flush enter 2024-05-15T13:20:56.651 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.651+0000 7f0c9b1c6640 10 librados: watch_flush exit 2024-05-15T13:20:56.651 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.651+0000 7f0c9b1c6640 20 client.16125.objecter shutdown clearing up homeless session... 2024-05-15T13:20:56.651 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.651+0000 7f0c9b1c6640 10 client.16125.objecter successfully canceled tick 2024-05-15T13:20:56.652 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.652+0000 7f0c9b1c6640 1 -- 172.21.15.117:0/3544704919 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f0c8006ce90 msgr2=0x7f0c8006f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:56.652 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.652+0000 7f0c9b1c6640 1 --2- 172.21.15.117:0/3544704919 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f0c8006ce90 0x7f0c8006f350 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f0c88035000 tx=0x7f0c88030000 comp rx=0 tx=0).stop 2024-05-15T13:20:56.652 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.652+0000 7f0c9b1c6640 10 monclient: shutdown 2024-05-15T13:20:56.652 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.652+0000 7f0c9b1c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:20:56.652 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.652+0000 7f0c9b1c6640 1 -- 172.21.15.117:0/3544704919 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c9406c900 msgr2=0x7f0c94145f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:20:56.652 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.652+0000 7f0c9b1c6640 1 --2- 172.21.15.117:0/3544704919 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c9406c900 0x7f0c94145f50 secure :-1 s=READY pgs=3316 cs=0 l=1 rev1=1 crypto rx=0x7f0c8400c400 tx=0x7f0c8400c8c0 comp rx=0 tx=0).stop 2024-05-15T13:20:56.652 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.652+0000 7f0c9b1c6640 1 -- 172.21.15.117:0/3544704919 shutdown_connections 2024-05-15T13:20:56.652 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.652+0000 7f0c9b1c6640 1 --2- 172.21.15.117:0/3544704919 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0c941431b0 0x7f0c941d7840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:56.652 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.652+0000 7f0c9b1c6640 1 --2- 172.21.15.117:0/3544704919 >> [v2:172.21.15.117:6800/1785766523,v1:172.21.15.117:6801/1785766523] conn(0x7f0c8006ce90 0x7f0c8006f350 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:56.652 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.652+0000 7f0c9b1c6640 1 --2- 172.21.15.117:0/3544704919 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0c940af1c0 0x7f0c9411f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:56.652 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.652+0000 7f0c9b1c6640 1 --2- 172.21.15.117:0/3544704919 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c9406c900 0x7f0c94145f50 unknown :-1 s=CLOSED pgs=3316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:20:56.652 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.652+0000 7f0c9b1c6640 1 -- 172.21.15.117:0/3544704919 >> 172.21.15.117:0/3544704919 conn(0x7f0c940b2ac0 msgr2=0x7f0c940aea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:20:56.652 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.652+0000 7f0c9b1c6640 1 -- 172.21.15.117:0/3544704919 shutdown_connections 2024-05-15T13:20:56.652 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.652+0000 7f0c9b1c6640 1 -- 172.21.15.117:0/3544704919 wait complete. 2024-05-15T13:20:56.652 INFO:teuthology.orchestra.run.smithi117.stderr:2024-05-15T13:20:56.652+0000 7f0c9b1c6640 1 librados: shutdown 2024-05-15T13:20:56.663 INFO:teuthology.misc:Shutting down mds daemons... 2024-05-15T13:20:56.663 INFO:teuthology.misc:Shutting down osd daemons... 2024-05-15T13:20:56.663 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-05-15T13:20:56.663 INFO:teuthology.orchestra.run:waiting for 300 2024-05-15T13:20:56.681 INFO:tasks.ceph.osd.0.smithi117.stderr:2024-05-15T13:20:56.670+0000 b485640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 34623) UID: 0 2024-05-15T13:20:56.684 INFO:tasks.ceph.osd.0.smithi117.stderr:2024-05-15T13:20:56.672+0000 b485640 -1 osd.0 1225 *** Got signal Terminated *** 2024-05-15T13:21:02.033 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:39:09.203 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:21:02.036 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:39:09.206 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:21:02.053 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:39:09.223 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:21:02.077 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:39:09.240 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:21:02.080 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:39:09.243 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:21:02.086 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:39:09.256 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:21:02.089 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:39:09.259 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:21:02.098 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:39:09.261 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:21:02.108 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:39:09.278 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:21:02.795 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:39:09.958 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:21:02.796 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:39:09.959 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:21:02.804 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:39:09.967 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:21:02.825 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:39:09.995 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:21:02.826 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:39:09.996 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:21:02.834 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:39:10.004 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:21:02.853 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:39:10.023 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:21:02.854 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:39:10.024 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:21:02.862 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:39:10.032 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:21:07.635 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-05-15T13:21:08.866 INFO:tasks.ceph.osd.0:Stopped 2024-05-15T13:21:08.867 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-05-15T13:21:08.867 INFO:teuthology.orchestra.run:waiting for 300 2024-05-15T13:21:08.879 INFO:tasks.ceph.osd.1.smithi117.stderr:2024-05-15T13:21:08.872+0000 b485640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 34631) UID: 0 2024-05-15T13:21:08.879 INFO:tasks.ceph.osd.1.smithi117.stderr:2024-05-15T13:21:08.873+0000 b485640 -1 osd.1 1227 *** Got signal Terminated *** 2024-05-15T13:21:13.846 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-05-15T13:21:22.070 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:39:29.240 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:21:22.074 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:39:29.244 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:21:22.102 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:39:29.272 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:21:22.124 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:39:29.286 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:21:22.124 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:39:29.294 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:21:22.126 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:39:29.289 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:21:22.127 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:39:29.296 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:21:22.147 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:39:29.310 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:21:22.148 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:39:29.318 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:21:22.838 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:39:30.001 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:21:22.839 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:39:30.002 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:21:22.847 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:39:30.010 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:21:22.904 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:39:30.074 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:21:22.905 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:39:30.075 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:21:22.906 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:39:30.076 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:21:22.910 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:39:30.077 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:21:22.915 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:39:30.085 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:21:22.919 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:39:30.088 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:21:32.378 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-05-15T13:21:33.273 INFO:tasks.ceph.osd.1:Stopped 2024-05-15T13:21:33.273 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-05-15T13:21:33.273 INFO:teuthology.orchestra.run:waiting for 300 2024-05-15T13:21:33.286 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-05-15T13:21:33.278+0000 b485640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 34638) UID: 0 2024-05-15T13:21:33.286 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-05-15T13:21:33.279+0000 b485640 -1 osd.2 1229 *** Got signal Terminated *** 2024-05-15T13:21:38.386 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-05-15T13:21:38.386 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-05-15T13:21:39.375 INFO:tasks.ceph.osd.2:Stopped 2024-05-15T13:21:39.376 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-05-15T13:21:39.376 INFO:teuthology.orchestra.run:waiting for 300 2024-05-15T13:21:39.389 INFO:tasks.ceph.osd.3.smithi117.stderr:2024-05-15T13:21:39.381+0000 b485640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 34644) UID: 0 2024-05-15T13:21:39.391 INFO:tasks.ceph.osd.3.smithi117.stderr:2024-05-15T13:21:39.382+0000 b485640 -1 osd.3 1231 *** Got signal Terminated *** 2024-05-15T13:21:42.158 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:39:49.326 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:21:42.162 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:39:49.332 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:21:42.182 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:39:49.352 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:21:42.211 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:39:49.373 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:21:42.212 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:39:49.375 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:21:42.214 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:39:49.383 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:21:42.216 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:39:49.386 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:21:42.230 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:39:49.393 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:21:42.235 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:39:49.405 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:21:42.922 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:39:50.085 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:21:42.923 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:39:50.086 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:21:42.931 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:39:50.094 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:21:42.962 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:39:50.131 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:21:42.963 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:39:50.132 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:21:42.966 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:39:50.135 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:21:42.967 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:39:50.137 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:21:42.972 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:39:50.141 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:21:42.976 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:39:50.146 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:21:44.293 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-05-15T13:21:44.293 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-05-15T13:21:45.478 INFO:tasks.ceph.osd.3:Stopped 2024-05-15T13:21:45.479 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-05-15T13:21:45.479 INFO:teuthology.orchestra.run:waiting for 300 2024-05-15T13:21:45.493 INFO:tasks.ceph.osd.4.smithi178.stderr:2024-05-15T13:21:45.485+0000 b485640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 34474) UID: 0 2024-05-15T13:21:45.495 INFO:tasks.ceph.osd.4.smithi178.stderr:2024-05-15T13:21:45.486+0000 b485640 -1 osd.4 1233 *** Got signal Terminated *** 2024-05-15T13:21:50.101 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2024-05-15T13:21:50.101 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-05-15T13:21:51.581 INFO:tasks.ceph.osd.4:Stopped 2024-05-15T13:21:51.581 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-05-15T13:21:51.582 INFO:teuthology.orchestra.run:waiting for 300 2024-05-15T13:21:51.595 INFO:tasks.ceph.osd.5.smithi178.stderr:2024-05-15T13:21:51.587+0000 b485640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 34478) UID: 0 2024-05-15T13:21:51.599 INFO:tasks.ceph.osd.5.smithi178.stderr:2024-05-15T13:21:51.588+0000 b485640 -1 osd.5 1235 *** Got signal Terminated *** 2024-05-15T13:21:55.909 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2024-05-15T13:22:02.229 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:40:09.398 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:22:02.231 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:40:09.401 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:22:02.255 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:40:09.424 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:22:02.296 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:40:09.459 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:22:02.298 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:40:09.468 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:22:02.299 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:40:09.461 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:22:02.304 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:40:09.473 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:22:02.318 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:40:09.480 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:22:02.325 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:40:09.495 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:22:03.101 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:40:10.263 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:22:03.101 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:40:10.264 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:22:03.109 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:40:10.272 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:22:03.133 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:40:10.303 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:22:03.134 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:40:10.303 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:22:03.134 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:40:10.304 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:22:03.135 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:40:10.304 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:22:03.142 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:40:10.312 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:22:03.143 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:40:10.312 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:22:22.300 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:40:29.469 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:22:22.304 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:40:29.473 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:22:22.322 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:40:29.491 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:22:22.358 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:40:29.527 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:22:22.359 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:40:29.528 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:22:22.361 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:40:29.523 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:22:22.363 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:40:29.526 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:22:22.377 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:40:29.547 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:22:22.379 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:40:29.541 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:22:23.153 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:40:30.315 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:22:23.154 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:40:30.317 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:22:23.162 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:40:30.325 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:22:23.182 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:40:30.351 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:22:23.183 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:40:30.353 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:22:23.185 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:40:30.354 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:22:23.186 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:40:30.355 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:22:23.191 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:40:30.361 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:22:23.194 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:40:30.363 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:22:42.368 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:40:49.537 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:22:42.371 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:40:49.540 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:22:42.395 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:40:49.564 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:22:42.419 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:40:49.581 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:22:42.422 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:40:49.584 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:22:42.429 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:40:49.598 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:22:42.432 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:40:49.601 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:22:42.440 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:40:49.602 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:22:42.466 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:40:49.635 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:22:43.145 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:40:50.307 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:22:43.146 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:40:50.308 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:22:43.151 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:40:50.320 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:22:43.152 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:40:50.321 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:22:43.154 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:40:50.316 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:22:43.160 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:40:50.329 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:22:43.173 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:40:50.341 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:22:43.174 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:40:50.342 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:22:43.181 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:40:50.350 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:22:48.103 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2024-05-15T13:22:52.596 INFO:tasks.ceph.osd.5:Stopped 2024-05-15T13:22:52.596 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-05-15T13:22:52.596 INFO:teuthology.orchestra.run:waiting for 300 2024-05-15T13:22:52.610 INFO:tasks.ceph.osd.6.smithi178.stderr:2024-05-15T13:22:52.601+0000 b485640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 34481) UID: 0 2024-05-15T13:22:52.610 INFO:tasks.ceph.osd.6.smithi178.stderr:2024-05-15T13:22:52.602+0000 b485640 -1 osd.6 1237 *** Got signal Terminated *** 2024-05-15T13:22:53.810 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2024-05-15T13:23:02.433 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:41:09.602 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:23:02.435 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:41:09.604 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:23:02.458 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:41:09.626 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:23:02.478 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:41:09.640 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:23:02.481 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:41:09.643 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:23:02.488 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:41:09.657 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:23:02.508 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:41:09.670 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:23:02.511 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:41:09.680 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:23:02.530 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:41:09.699 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:23:03.394 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:41:10.556 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:23:03.396 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:41:10.558 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:23:03.413 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:41:10.575 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:23:03.590 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:41:10.759 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:23:03.592 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:41:10.761 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:23:03.605 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:41:10.774 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:23:03.674 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:41:10.843 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:23:03.679 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:41:10.848 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:23:03.748 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:41:10.917 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:23:17.002 INFO:tasks.ceph.osd.6:Stopped 2024-05-15T13:23:17.002 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-05-15T13:23:17.002 INFO:teuthology.orchestra.run:waiting for 300 2024-05-15T13:23:17.017 INFO:tasks.ceph.osd.7.smithi178.stderr:2024-05-15T13:23:17.009+0000 b485640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 34482) UID: 0 2024-05-15T13:23:17.017 INFO:tasks.ceph.osd.7.smithi178.stderr:2024-05-15T13:23:17.010+0000 b485640 -1 osd.7 1237 *** Got signal Terminated *** 2024-05-15T13:23:22.139 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2024-05-15T13:23:22.471 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:41:29.639 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:23:22.473 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:41:29.642 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:23:22.496 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:41:29.665 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:23:22.526 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:41:29.688 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:23:22.527 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:41:29.689 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:23:22.531 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:41:29.700 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:23:22.532 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:41:29.701 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:23:22.549 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:41:29.711 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:23:22.554 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:41:29.722 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:23:23.071 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:41:30.233 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:23:23.072 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:41:30.234 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:23:23.080 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:41:30.242 33941== Warning: unimplemented fcntl command: 1036 2024-05-15T13:23:23.098 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:41:30.264 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:23:23.098 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:41:30.265 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:23:23.103 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:41:30.272 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:23:23.103 INFO:tasks.ceph.osd.7:Stopped 2024-05-15T13:23:23.103 INFO:teuthology.misc:Shutting down mgr daemons... 2024-05-15T13:23:23.104 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-05-15T13:23:23.104 INFO:teuthology.orchestra.run:waiting for 300 2024-05-15T13:23:23.104 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:41:30.273 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:23:23.105 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:41:30.273 34254== Warning: unimplemented fcntl command: 1036 2024-05-15T13:23:23.105 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-05-15T13:23:23.105+0000 7f5130389640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i y (PID: 34268) UID: 0 2024-05-15T13:23:23.105 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-05-15T13:23:23.105+0000 7f5130389640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-05-15T13:23:23.117 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:41:30.286 34258== Warning: unimplemented fcntl command: 1036 2024-05-15T13:23:23.136 INFO:tasks.ceph.mgr.y:Stopped 2024-05-15T13:23:23.136 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-05-15T13:23:23.136 INFO:teuthology.orchestra.run:waiting for 300 2024-05-15T13:23:23.161 INFO:tasks.ceph.mgr.x:Stopped 2024-05-15T13:23:23.161 INFO:teuthology.misc:Shutting down mon daemons... 2024-05-15T13:23:23.161 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-05-15T13:23:23.161 INFO:teuthology.orchestra.run:waiting for 300 2024-05-15T13:23:23.273 INFO:tasks.ceph.mon.a.smithi117.stderr:2024-05-15T13:23:23.257+0000 9f99640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 34229) UID: 0 2024-05-15T13:23:23.274 INFO:tasks.ceph.mon.a.smithi117.stderr:2024-05-15T13:23:23.259+0000 9f99640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-05-15T13:23:24.392 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:41:31.560 34254== 2024-05-15T13:23:24.392 INFO:tasks.ceph.mon.a.smithi117.stderr:==00:00:41:31.560 34254== Exit program on first error (--exit-on-first-error=yes) 2024-05-15T13:23:24.415 INFO:tasks.ceph.mon.a.smithi117.stderr:daemon-helper: command failed with exit status 42 2024-05-15T13:23:27.342 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2024-05-15T13:23:27.342 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2024-05-15T13:23:29.263 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-05-15T13:23:29.263 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi117 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-05-15T13:23:29.264 INFO:tasks.ceph.mon.a:Stopped 2024-05-15T13:23:29.265 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-05-15T13:23:29.265 INFO:teuthology.orchestra.run:waiting for 300 2024-05-15T13:23:29.280 INFO:tasks.ceph.mon.c.smithi117.stderr:2024-05-15T13:23:29.269+0000 9f99640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 34231) UID: 0 2024-05-15T13:23:29.281 INFO:tasks.ceph.mon.c.smithi117.stderr:2024-05-15T13:23:29.271+0000 9f99640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-05-15T13:23:29.991 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:41:37.159 34258== 2024-05-15T13:23:29.991 INFO:tasks.ceph.mon.c.smithi117.stderr:==00:00:41:37.160 34258== Exit program on first error (--exit-on-first-error=yes) 2024-05-15T13:23:30.010 INFO:tasks.ceph.mon.c.smithi117.stderr:daemon-helper: command failed with exit status 42 2024-05-15T13:23:32.444 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2024-05-15T13:23:32.444 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2024-05-15T13:23:35.366 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-05-15T13:23:35.366 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi117 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-05-15T13:23:35.366 INFO:tasks.ceph.mon.c:Stopped 2024-05-15T13:23:35.367 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-05-15T13:23:35.367 INFO:teuthology.orchestra.run:waiting for 300 2024-05-15T13:23:35.384 INFO:tasks.ceph.mon.b.smithi178.stderr:2024-05-15T13:23:35.371+0000 9f99640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 33936) UID: 0 2024-05-15T13:23:35.385 INFO:tasks.ceph.mon.b.smithi178.stderr:2024-05-15T13:23:35.373+0000 9f99640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2024-05-15T13:23:35.944 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:41:43.105 33941== 2024-05-15T13:23:35.944 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:41:43.105 33941== Exit program on first error (--exit-on-first-error=yes) 2024-05-15T13:23:35.963 INFO:tasks.ceph.mon.b.smithi178.stderr:daemon-helper: command failed with exit status 42 2024-05-15T13:23:37.445 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2024-05-15T13:23:37.445 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2024-05-15T13:23:41.468 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-05-15T13:23:41.468 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi178 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-05-15T13:23:41.468 INFO:tasks.ceph.mon.b:Stopped 2024-05-15T13:23:41.469 INFO:tasks.ceph:Checking cluster log for badness... 2024-05-15T13:23:41.469 DEBUG:teuthology.orchestra.run.smithi117:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-05-15T13:23:41.506 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi117.front.sepia.ceph.com 2024-05-15T13:23:41.507 DEBUG:teuthology.orchestra.run.smithi117:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-05-15T13:23:41.721 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi117.front.sepia.ceph.com 2024-05-15T13:23:41.721 DEBUG:teuthology.orchestra.run.smithi117:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-05-15T13:23:41.820 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi117.front.sepia.ceph.com 2024-05-15T13:23:41.820 DEBUG:teuthology.orchestra.run.smithi117:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-05-15T13:23:41.895 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi117.front.sepia.ceph.com 2024-05-15T13:23:41.895 DEBUG:teuthology.orchestra.run.smithi117:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-05-15T13:23:41.953 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi178.front.sepia.ceph.com 2024-05-15T13:23:41.954 DEBUG:teuthology.orchestra.run.smithi178:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-05-15T13:23:42.021 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi178.front.sepia.ceph.com 2024-05-15T13:23:42.022 DEBUG:teuthology.orchestra.run.smithi178:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-05-15T13:23:42.076 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi178.front.sepia.ceph.com 2024-05-15T13:23:42.077 DEBUG:teuthology.orchestra.run.smithi178:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-05-15T13:23:42.142 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi178.front.sepia.ceph.com 2024-05-15T13:23:42.142 DEBUG:teuthology.orchestra.run.smithi178:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-05-15T13:23:42.219 INFO:tasks.ceph:Archiving mon data... 2024-05-15T13:23:42.220 DEBUG:teuthology.misc:Transferring archived files from smithi117:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707111/data/mon.a.tgz 2024-05-15T13:23:42.220 DEBUG:teuthology.orchestra.run.smithi117:> mktemp 2024-05-15T13:23:42.236 INFO:teuthology.orchestra.run.smithi117.stdout:/tmp/tmp.zUtexVOMQb 2024-05-15T13:23:42.236 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.zUtexVOMQb 2024-05-15T13:23:42.407 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0666 /tmp/tmp.zUtexVOMQb 2024-05-15T13:23:42.446 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2024-05-15T13:23:42.491 DEBUG:teuthology.orchestra.remote:smithi117:/tmp/tmp.zUtexVOMQb is 568KB 2024-05-15T13:23:42.592 DEBUG:teuthology.orchestra.run.smithi117:> rm -fr /tmp/tmp.zUtexVOMQb 2024-05-15T13:23:42.607 DEBUG:teuthology.misc:Transferring archived files from smithi117:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707111/data/mon.c.tgz 2024-05-15T13:23:42.607 DEBUG:teuthology.orchestra.run.smithi117:> mktemp 2024-05-15T13:23:42.662 INFO:teuthology.orchestra.run.smithi117.stdout:/tmp/tmp.TV3BFWJINA 2024-05-15T13:23:42.662 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.TV3BFWJINA 2024-05-15T13:23:42.824 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0666 /tmp/tmp.TV3BFWJINA 2024-05-15T13:23:42.906 DEBUG:teuthology.orchestra.remote:smithi117:/tmp/tmp.TV3BFWJINA is 569KB 2024-05-15T13:23:43.004 DEBUG:teuthology.orchestra.run.smithi117:> rm -fr /tmp/tmp.TV3BFWJINA 2024-05-15T13:23:43.019 DEBUG:teuthology.misc:Transferring archived files from smithi178:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707111/data/mon.b.tgz 2024-05-15T13:23:43.020 DEBUG:teuthology.orchestra.run.smithi178:> mktemp 2024-05-15T13:23:43.038 INFO:teuthology.orchestra.run.smithi178.stdout:/tmp/tmp.vgHrOzIqeN 2024-05-15T13:23:43.039 DEBUG:teuthology.orchestra.run.smithi178:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.vgHrOzIqeN 2024-05-15T13:23:43.223 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod 0666 /tmp/tmp.vgHrOzIqeN 2024-05-15T13:23:43.307 DEBUG:teuthology.orchestra.remote:smithi178:/tmp/tmp.vgHrOzIqeN is 568KB 2024-05-15T13:23:43.379 DEBUG:teuthology.orchestra.run.smithi178:> rm -fr /tmp/tmp.vgHrOzIqeN 2024-05-15T13:23:43.395 INFO:tasks.ceph:Cleaning ceph cluster... 2024-05-15T13:23:43.395 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-05-15T13:23:43.397 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-05-15T13:23:43.749 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-05-15T13:23:43.986 INFO:tasks.ceph:Archiving crash dumps... 2024-05-15T13:23:43.987 DEBUG:teuthology.misc:Transferring archived files from smithi117:/var/lib/ceph/crash to /home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707111/remote/smithi117/crash 2024-05-15T13:23:43.988 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-05-15T13:23:44.013 DEBUG:teuthology.misc:Transferring archived files from smithi178:/var/lib/ceph/crash to /home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707111/remote/smithi178/crash 2024-05-15T13:23:44.014 DEBUG:teuthology.orchestra.run.smithi178:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-05-15T13:23:44.038 INFO:tasks.ceph:Compressing logs... 2024-05-15T13:23:44.039 DEBUG:teuthology.orchestra.run.smithi117:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-15T13:23:44.059 DEBUG:teuthology.orchestra.run.smithi178:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-15T13:23:44.095 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38973.log 2024-05-15T13:23:44.096 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41125.log 2024-05-15T13:23:44.096 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107753.log 2024-05-15T13:23:44.096 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.41125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40830.log 2024-05-15T13:23:44.096 INFO:teuthology.orchestra.run.smithi117.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38973.log.gz 2024-05-15T13:23:44.096 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.107753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40372.log 2024-05-15T13:23:44.096 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41098.log 2024-05-15T13:23:44.096 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40830.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.107753.log.gz 2024-05-15T13:23:44.096 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34271.log 2024-05-15T13:23:44.096 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.41098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67194.log 2024-05-15T13:23:44.096 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.34271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41114.log 2024-05-15T13:23:44.096 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.67194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38720.log 2024-05-15T13:23:44.096 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.41114.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.34271.log.gz 2024-05-15T13:23:44.096 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107932.log 2024-05-15T13:23:44.097 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85532.log 2024-05-15T13:23:44.097 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.107932.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.38720.log.gz 2024-05-15T13:23:44.097 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42378.log 91.1% -- replaced with /var/log/ceph/ceph-client.admin.41098.log.gz 2024-05-15T13:23:44.097 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:23:44.097 INFO:teuthology.orchestra.run.smithi117.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.67194.log.gz 2024-05-15T13:23:44.097 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.85532.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.107932.log.gz 2024-05-15T13:23:44.097 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39002.log 2024-05-15T13:23:44.097 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.42378.log: 91.2%gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-05-15T13:23:44.097 INFO:teuthology.orchestra.run.smithi117.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40830.log.gz 2024-05-15T13:23:44.097 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108186.log 2024-05-15T13:23:44.097 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85531.log 2024-05-15T13:23:44.097 INFO:teuthology.orchestra.run.smithi117.stderr: 87.6% 91.0% -- replaced with /var/log/ceph/ceph-client.admin.42378.log.gz 2024-05-15T13:23:44.097 INFO:teuthology.orchestra.run.smithi117.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39002.log.gz 2024-05-15T13:23:44.097 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.108186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40420.log 2024-05-15T13:23:44.097 INFO:teuthology.orchestra.run.smithi117.stderr: 87.0%/var/log/ceph/ceph-client.admin.85531.log: -- replaced with /var/log/ceph/ceph-client.admin.108186.log.gz 2024-05-15T13:23:44.097 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40560.log 2024-05-15T13:23:44.097 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 90.5% -- replaced with /var/log/ceph/ceph-client.admin.85532.log.gz 2024-05-15T13:23:44.098 INFO:teuthology.orchestra.run.smithi117.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39078.log 2024-05-15T13:23:44.098 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40560.log: /var/log/ceph/ceph-client.admin.40420.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-05-15T13:23:44.098 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39078.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39078.log.gz 90.6% -- replaced with /var/log/ceph/ceph-client.admin.85531.log.gz 2024-05-15T13:23:44.098 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:23:44.098 INFO:teuthology.orchestra.run.smithi117.stderr: 91.9% -- replaced with /var/log/ceph/ceph-client.admin.40372.log.gz 2024-05-15T13:23:44.098 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39551.log 2024-05-15T13:23:44.098 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39257.log 2024-05-15T13:23:44.098 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40232.log 2024-05-15T13:23:44.098 INFO:teuthology.orchestra.run.smithi117.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.39551.log.gz 2024-05-15T13:23:44.098 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67195.log 2024-05-15T13:23:44.098 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40232.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.39257.log.gz 2024-05-15T13:23:44.098 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108017.log 2024-05-15T13:23:44.098 INFO:teuthology.orchestra.run.smithi117.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.40420.log.gz 2024-05-15T13:23:44.098 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.67195.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.41114.log.gz 2024-05-15T13:23:44.108 INFO:teuthology.orchestra.run.smithi117.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.67195.log.gz 2024-05-15T13:23:44.108 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42635.log 2024-05-15T13:23:44.108 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.108017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107711.log 2024-05-15T13:23:44.108 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.42635.log: gzip -5 --verbose 87.5% -- replaced with /var/log/ceph/ceph-client.admin.108017.log.gz 2024-05-15T13:23:44.108 INFO:teuthology.orchestra.run.smithi117.stderr: -- /var/log/ceph/ceph.audit.log 2024-05-15T13:23:44.108 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38377.log 2024-05-15T13:23:44.109 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-client.admin.107711.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.107711.log.gz 2024-05-15T13:23:44.109 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107333.log 2024-05-15T13:23:44.109 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38377.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.42635.log.gz 2024-05-15T13:23:44.109 INFO:teuthology.orchestra.run.smithi117.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.38377.log.gz 2024-05-15T13:23:44.109 INFO:teuthology.orchestra.run.smithi117.stderr: 92.7% -- replaced with /var/log/ceph/ceph.log.gz 2024-05-15T13:23:44.109 INFO:teuthology.orchestra.run.smithi117.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.40232.log.gz 2024-05-15T13:23:44.109 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40006.log 2024-05-15T13:23:44.109 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34679.log 2024-05-15T13:23:44.109 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.107333.log: /var/log/ceph/ceph-client.admin.40006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38993.log 2024-05-15T13:23:44.109 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.34679.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.107333.log.gz 2024-05-15T13:23:44.109 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40795.log 2024-05-15T13:23:44.109 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38993.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34679.log.gz 2024-05-15T13:23:44.109 INFO:teuthology.orchestra.run.smithi117.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.38993.log.gz 2024-05-15T13:23:44.109 INFO:teuthology.orchestra.run.smithi117.stderr: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.40006.log.gz 2024-05-15T13:23:44.110 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40669.log 2024-05-15T13:23:44.110 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107417.log 2024-05-15T13:23:44.110 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40669.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-05-15T13:23:44.110 INFO:teuthology.orchestra.run.smithi117.stderr: 89.8%/var/log/ceph/ceph-client.admin.107417.log: -- replaced with /var/log/ceph/ceph-client.admin.40795.log.gz 2024-05-15T13:23:44.110 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41108.log 2024-05-15T13:23:44.110 INFO:teuthology.orchestra.run.smithi117.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40669.log.gz 2024-05-15T13:23:44.110 INFO:teuthology.orchestra.run.smithi117.stderr:gzip 88.1% -5 --verbose -- /var/log/ceph/ceph-client.admin.39970.log 2024-05-15T13:23:44.110 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-osd.3.log: -- replaced with /var/log/ceph/ceph-client.admin.107417.log.gz 2024-05-15T13:23:44.110 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.41108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34649.log 2024-05-15T13:23:44.110 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39980.log 2024-05-15T13:23:44.110 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39970.log: /var/log/ceph/ceph-client.admin.34649.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.34649.log.gz 2024-05-15T13:23:44.110 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107543.log 2024-05-15T13:23:44.123 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34386.log 2024-05-15T13:23:44.124 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-05-15T13:23:44.124 INFO:teuthology.orchestra.run.smithi178.stderr:gzip/var/log/ceph/ceph-client.admin.34386.log: -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-05-15T13:23:44.124 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-05-15T13:23:44.124 INFO:teuthology.orchestra.run.smithi178.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34386.log.gz 2024-05-15T13:23:44.124 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-05-15T13:23:44.124 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-05-15T13:23:44.124 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34094.log 2024-05-15T13:23:44.124 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34346.log 2024-05-15T13:23:44.124 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34094.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-05-15T13:23:44.124 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34346.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-05-15T13:23:44.124 INFO:teuthology.orchestra.run.smithi178.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.34094.log.gz 2024-05-15T13:23:44.124 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-mgr.x.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34346.log.gz 2024-05-15T13:23:44.124 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-05-15T13:23:44.124 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-05-15T13:23:44.124 INFO:teuthology.orchestra.run.smithi178.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz/var/log/ceph/valgrind/osd.6.log: 2024-05-15T13:23:44.125 INFO:teuthology.orchestra.run.smithi178.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-05-15T13:23:44.125 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-05-15T13:23:44.125 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/valgrind/osd.4.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-05-15T13:23:44.125 INFO:teuthology.orchestra.run.smithi178.stderr: 63.9% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-05-15T13:23:44.125 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/valgrind/osd.7.log: 63.9% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-05-15T13:23:44.125 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34305.log 2024-05-15T13:23:44.125 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/valgrind/osd.5.log: 63.9% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-05-15T13:23:44.125 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-05-15T13:23:44.125 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34185.log 2024-05-15T13:23:44.125 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-osd.6.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34305.log.gz 2024-05-15T13:23:44.125 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34145.log 2024-05-15T13:23:44.125 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34185.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.34185.log.gz 2024-05-15T13:23:44.125 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34225.log 86.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-05-15T13:23:44.125 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-05-15T13:23:44.125 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34265.log 2024-05-15T13:23:44.126 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34225.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.34145.log.gz 2024-05-15T13:23:44.126 INFO:teuthology.orchestra.run.smithi178.stderr: 92.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-05-15T13:23:44.126 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-05-15T13:23:44.126 INFO:teuthology.orchestra.run.smithi178.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34225.log.gz 2024-05-15T13:23:44.126 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34265.log: /var/log/ceph/ceph-mon.b.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34265.log.gz 2024-05-15T13:23:44.128 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39980.log: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.39970.log.gz 2024-05-15T13:23:44.128 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41018.log 2024-05-15T13:23:44.128 INFO:teuthology.orchestra.run.smithi117.stderr: 95.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-05-15T13:23:44.128 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.107543.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.107543.log.gz 2024-05-15T13:23:44.128 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40102.log 2024-05-15T13:23:44.129 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.41018.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 92.2% -- replaced with /var/log/ceph/ceph-client.admin.39980.log.gz 2024-05-15T13:23:44.129 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:23:44.129 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.40102.log: -- /var/log/ceph/ceph-client.admin.38719.log 2024-05-15T13:23:44.129 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-osd.1.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.41108.log.gz 2024-05-15T13:23:44.129 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-05-15T13:23:44.129 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38719.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38719.log.gz 2024-05-15T13:23:44.129 INFO:teuthology.orchestra.run.smithi117.stderr: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.40102.log.gz 2024-05-15T13:23:44.129 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40389.log 2024-05-15T13:23:44.129 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-05-15T13:23:44.131 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40389.log: 91.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40734.log 2024-05-15T13:23:44.131 INFO:teuthology.orchestra.run.smithi117.stderr: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.41125.log.gz 2024-05-15T13:23:44.131 INFO:teuthology.orchestra.run.smithi117.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40389.log.gz 2024-05-15T13:23:44.132 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39155.log 2024-05-15T13:23:44.132 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40734.log: 93.9% -- replaced with /var/log/ceph/ceph-client.admin.41018.log.gz 2024-05-15T13:23:44.132 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38417.log 2024-05-15T13:23:44.134 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39155.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.39155.log.gz 2024-05-15T13:23:44.134 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40750.log 2024-05-15T13:23:44.135 INFO:teuthology.orchestra.run.smithi178.stderr: 94.9% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-05-15T13:23:44.136 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38417.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.40734.log.gz 2024-05-15T13:23:44.137 INFO:teuthology.orchestra.run.smithi117.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.38417.log.gz 2024-05-15T13:23:44.137 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107975.log 2024-05-15T13:23:44.137 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38709.log 2024-05-15T13:23:44.137 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.107975.log: /var/log/ceph/ceph-client.admin.40750.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.107975.log.gz 2024-05-15T13:23:44.137 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74321.log 2024-05-15T13:23:44.137 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38709.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38709.log.gz 2024-05-15T13:23:44.138 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38721.log 2024-05-15T13:23:44.139 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.74321.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.40750.log.gz 2024-05-15T13:23:44.139 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107501.log 2024-05-15T13:23:44.151 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38721.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38721.log.gz 2024-05-15T13:23:44.151 INFO:teuthology.orchestra.run.smithi117.stderr: 92.1% -- replaced with /var/log/ceph/ceph-client.admin.40560.log.gz 2024-05-15T13:23:44.151 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-05-15T13:23:44.151 INFO:teuthology.orchestra.run.smithi117.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.74321.log.gzgzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-05-15T13:23:44.152 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:23:44.152 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.107501.log: /var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-05-15T13:23:44.152 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/valgrind/osd.0.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.107501.log.gz 2024-05-15T13:23:44.152 INFO:teuthology.orchestra.run.smithi117.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-05-15T13:23:44.152 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-05-15T13:23:44.152 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-05-15T13:23:44.152 INFO:teuthology.orchestra.run.smithi117.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-05-15T13:23:44.153 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/valgrind/osd.3.log: 63.9% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-05-15T13:23:44.153 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-05-15T13:23:44.153 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-05-15T13:23:44.153 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/valgrind/osd.1.log: /var/log/ceph/valgrind/mon.c.log: 64.0% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-05-15T13:23:44.153 INFO:teuthology.orchestra.run.smithi117.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-05-15T13:23:44.153 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38543.log 2024-05-15T13:23:44.154 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/valgrind/osd.2.log: 64.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-05-15T13:23:44.154 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37982.log 2024-05-15T13:23:44.154 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40433.log 2024-05-15T13:23:44.154 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38543.log: /var/log/ceph/ceph-client.admin.37982.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38543.log.gz 2024-05-15T13:23:44.155 INFO:teuthology.orchestra.run.smithi117.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.37982.log.gz 2024-05-15T13:23:44.155 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107837.log 2024-05-15T13:23:44.159 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40433.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.40433.log.gz 2024-05-15T13:23:44.159 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38716.log 2024-05-15T13:23:44.159 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38459.log 2024-05-15T13:23:44.159 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.107837.log: /var/log/ceph/ceph-client.admin.38716.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.107837.log.gz 2024-05-15T13:23:44.159 INFO:teuthology.orchestra.run.smithi117.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38716.log.gz 2024-05-15T13:23:44.160 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-05-15T13:23:44.160 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38459.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.38459.log.gz 2024-05-15T13:23:44.160 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41049.log 2024-05-15T13:23:44.160 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42634.log 2024-05-15T13:23:44.171 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-mon.a.log: /var/log/ceph/ceph-client.admin.41049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40752.log 2024-05-15T13:23:44.174 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.42634.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.42634.log.gz 2024-05-15T13:23:44.187 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39020.log 2024-05-15T13:23:44.190 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40752.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.40752.log.gz 2024-05-15T13:23:44.196 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40550.log 2024-05-15T13:23:44.197 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39020.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39020.log.gz 2024-05-15T13:23:44.201 INFO:teuthology.orchestra.run.smithi117.stderr: 96.5% -- replaced with /var/log/ceph/ceph-client.admin.41049.log.gz 2024-05-15T13:23:44.201 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40973.log 2024-05-15T13:23:44.210 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40550.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.40550.log.gz 2024-05-15T13:23:44.211 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74320.log 2024-05-15T13:23:44.224 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40973.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.40973.log.gz 2024-05-15T13:23:44.224 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39397.log 2024-05-15T13:23:44.226 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.74320.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.74320.log.gz 2024-05-15T13:23:44.228 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40247.log 2024-05-15T13:23:44.228 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39397.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.39397.log.gz 2024-05-15T13:23:44.228 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41050.log 2024-05-15T13:23:44.234 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40247.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.40247.log.gz 2024-05-15T13:23:44.234 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51803.log 2024-05-15T13:23:44.237 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.41050.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.41050.log.gz 2024-05-15T13:23:44.238 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40025.log 2024-05-15T13:23:44.242 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.51803.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.51803.log.gz 2024-05-15T13:23:44.242 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39635.log 2024-05-15T13:23:44.263 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40261.log 2024-05-15T13:23:44.263 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39635.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.39635.log.gz 2024-05-15T13:23:44.264 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85530.log 2024-05-15T13:23:44.273 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40261.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28408.log 2024-05-15T13:23:44.276 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.85530.log: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.40261.log.gz 2024-05-15T13:23:44.276 INFO:teuthology.orchestra.run.smithi117.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.85530.log.gz 2024-05-15T13:23:44.292 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107585.log 2024-05-15T13:23:44.292 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph.tmp-client.admin.28408.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28408.log.gz 2024-05-15T13:23:44.302 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67193.log 2024-05-15T13:23:44.302 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.107585.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.107585.log.gz 2024-05-15T13:23:44.309 INFO:teuthology.orchestra.run.smithi117.stderr: 92.8% -- replaced with /var/log/ceph/ceph-client.admin.40025.log.gz 2024-05-15T13:23:44.310 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40236.log 2024-05-15T13:23:44.313 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.67193.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.67193.log.gz 2024-05-15T13:23:44.313 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40778.log 2024-05-15T13:23:44.327 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40597.log 2024-05-15T13:23:44.343 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40778.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.40778.log.gz 2024-05-15T13:23:44.344 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38723.log 2024-05-15T13:23:44.357 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40597.log: 92.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40991.log 2024-05-15T13:23:44.358 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38723.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38723.log.gz 2024-05-15T13:23:44.359 INFO:teuthology.orchestra.run.smithi117.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40597.log.gz 2024-05-15T13:23:44.370 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107888.log 2024-05-15T13:23:44.383 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40175.log 2024-05-15T13:23:44.384 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.107888.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.107888.log.gz 2024-05-15T13:23:44.390 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38321.log 2024-05-15T13:23:44.400 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42379.log 2024-05-15T13:23:44.412 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38321.log: 87.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38911.log 2024-05-15T13:23:44.413 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.42379.log: -- replaced with /var/log/ceph/ceph-client.admin.38321.log.gz 2024-05-15T13:23:44.414 INFO:teuthology.orchestra.run.smithi117.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.42379.log.gz 2024-05-15T13:23:44.425 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39593.log 2024-05-15T13:23:44.426 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38911.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.38911.log.gz 2024-05-15T13:23:44.438 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40984.log 2024-05-15T13:23:44.439 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39593.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.39593.log.gz 2024-05-15T13:23:44.452 INFO:teuthology.orchestra.run.smithi117.stderr: 92.8% -- replaced with /var/log/ceph/ceph-client.admin.40991.log.gz 2024-05-15T13:23:44.465 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108226.log 2024-05-15T13:23:44.474 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40786.log 2024-05-15T13:23:44.483 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.108226.log: 88.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40044.log 2024-05-15T13:23:44.487 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40786.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.108226.log.gz 2024-05-15T13:23:44.499 INFO:teuthology.orchestra.run.smithi117.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40786.log.gz 2024-05-15T13:23:44.505 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38724.log 2024-05-15T13:23:44.510 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40044.log: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.40044.log.gz 2024-05-15T13:23:44.514 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40653.log 2024-05-15T13:23:44.519 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38724.log: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.40984.log.gz 2024-05-15T13:23:44.527 INFO:teuthology.orchestra.run.smithi117.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38724.log.gz 2024-05-15T13:23:44.530 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38725.log 2024-05-15T13:23:44.540 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40653.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.40653.log.gz 2024-05-15T13:23:44.552 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74319.log 2024-05-15T13:23:44.553 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38725.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38725.log.gz 2024-05-15T13:23:44.561 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39317.log 2024-05-15T13:23:44.568 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.74319.log: 94.2% -- replaced with /var/log/ceph/ceph-client.admin.40175.log.gz 2024-05-15T13:23:44.574 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40833.log 2024-05-15T13:23:44.576 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39317.log: 87.4% 90.6% -- replaced with /var/log/ceph/ceph-client.admin.74319.log.gz 2024-05-15T13:23:44.583 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40513.log 2024-05-15T13:23:44.584 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40833.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.39317.log.gz 2024-05-15T13:23:44.595 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107291.log 2024-05-15T13:23:44.597 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40513.log: -- replaced with /var/log/ceph/ceph-client.admin.40833.log.gz 2024-05-15T13:23:44.597 INFO:teuthology.orchestra.run.smithi117.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.40513.log.gz 2024-05-15T13:23:44.616 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39377.log 2024-05-15T13:23:44.617 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.107291.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.107291.log.gz 2024-05-15T13:23:44.625 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51804.log 2024-05-15T13:23:44.634 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40472.log 2024-05-15T13:23:44.638 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.51804.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.39377.log.gz 2024-05-15T13:23:44.643 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74322.log 2024-05-15T13:23:44.647 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40472.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.51804.log.gz 2024-05-15T13:23:44.652 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107669.log 2024-05-15T13:23:44.652 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.74322.log: 91.7% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-05-15T13:23:44.660 INFO:teuthology.orchestra.run.smithi117.stderr: 92.1% -- replaced with /var/log/ceph/ceph-client.admin.40472.log.gz 2024-05-15T13:23:44.664 INFO:teuthology.orchestra.run.smithi117.stderr: 90.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39927.log 2024-05-15T13:23:44.664 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.107669.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.107669.log.gz 2024-05-15T13:23:44.664 INFO:teuthology.orchestra.run.smithi117.stderr: -- replaced with /var/log/ceph/ceph-client.admin.74322.log.gz 2024-05-15T13:23:44.665 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39479.log 2024-05-15T13:23:44.681 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107627.log 2024-05-15T13:23:44.682 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39479.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39479.log.gz 2024-05-15T13:23:44.695 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35694.log 2024-05-15T13:23:44.697 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.107627.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.107627.log.gz 2024-05-15T13:23:44.697 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39297.log 2024-05-15T13:23:44.704 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.35694.log: 88.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108059.log 2024-05-15T13:23:44.704 INFO:teuthology.orchestra.run.smithi117.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35694.log.gz 2024-05-15T13:23:44.715 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39297.log: 88.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39946.log 2024-05-15T13:23:44.715 INFO:teuthology.orchestra.run.smithi117.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39297.log.gz 2024-05-15T13:23:44.716 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.108059.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.108059.log.gz 2024-05-15T13:23:44.717 INFO:teuthology.orchestra.run.smithi117.stderr: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.39927.log.gz 2024-05-15T13:23:44.718 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108143.log 2024-05-15T13:23:44.727 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34823.log 2024-05-15T13:23:44.728 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.108143.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.108143.log.gz 2024-05-15T13:23:44.741 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40849.log 2024-05-15T13:23:44.742 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.34823.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.34823.log.gz 2024-05-15T13:23:44.757 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38501.log 2024-05-15T13:23:44.758 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40849.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.40849.log.gz 2024-05-15T13:23:44.760 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42633.log 2024-05-15T13:23:44.760 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38501.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38501.log.gz 2024-05-15T13:23:44.772 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42377.log 2024-05-15T13:23:44.778 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.42633.log: 90.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38200.log 2024-05-15T13:23:44.785 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.42377.log: -- replaced with /var/log/ceph/ceph-client.admin.42633.log.gz 2024-05-15T13:23:44.794 INFO:teuthology.orchestra.run.smithi117.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.42377.log.gz 2024-05-15T13:23:44.798 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74318.log 2024-05-15T13:23:44.799 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38200.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.38200.log.gz 2024-05-15T13:23:44.807 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36776.log 2024-05-15T13:23:44.810 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.74318.log: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.39946.log.gz 2024-05-15T13:23:44.811 INFO:teuthology.orchestra.run.smithi117.stderr: 90.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40099.log 2024-05-15T13:23:44.813 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.36776.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.36776.log.gz 2024-05-15T13:23:44.813 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40272.log 2024-05-15T13:23:44.822 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40099.log: -- replaced with /var/log/ceph/ceph-client.admin.74318.log.gz 2024-05-15T13:23:44.826 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39437.log 2024-05-15T13:23:44.833 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107375.log 2024-05-15T13:23:44.840 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38272.log 2024-05-15T13:23:44.840 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.107375.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39437.log.gz 2024-05-15T13:23:44.846 INFO:teuthology.orchestra.run.smithi117.stderr: 87.8% 91.0% -- replaced with /var/log/ceph/ceph-client.admin.40272.log.gz 2024-05-15T13:23:44.854 INFO:teuthology.orchestra.run.smithi117.stderr: -- replaced with /var/log/ceph/ceph-client.admin.107375.log.gz 2024-05-15T13:23:44.859 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108101.log 2024-05-15T13:23:44.860 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38272.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38272.log.gz 2024-05-15T13:23:44.868 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107795.log 2024-05-15T13:23:44.877 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.108101.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.108101.log.gz 2024-05-15T13:23:44.879 INFO:teuthology.orchestra.run.smithi117.stderr: 93.2% -- replaced with /var/log/ceph/ceph-client.admin.40236.log.gz 2024-05-15T13:23:44.880 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107459.log 2024-05-15T13:23:44.882 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.107795.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.107795.log.gz 2024-05-15T13:23:44.882 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38992.log 2024-05-15T13:23:44.883 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.107459.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.107459.log.gz 2024-05-15T13:23:44.884 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38992.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38992.log.gz 2024-05-15T13:23:45.760 INFO:teuthology.orchestra.run.smithi178.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-05-15T13:23:45.818 INFO:teuthology.orchestra.run.smithi117.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-05-15T13:23:46.184 INFO:teuthology.orchestra.run.smithi117.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.40099.log.gz 2024-05-15T13:23:48.412 INFO:teuthology.orchestra.run.smithi117.stderr: 90.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-05-15T13:23:50.299 INFO:teuthology.orchestra.run.smithi178.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-05-15T13:23:50.345 INFO:teuthology.orchestra.run.smithi178.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-05-15T13:23:50.896 INFO:teuthology.orchestra.run.smithi178.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-05-15T13:23:51.039 INFO:teuthology.orchestra.run.smithi117.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-05-15T13:23:51.376 INFO:teuthology.orchestra.run.smithi117.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-05-15T13:23:51.394 INFO:teuthology.orchestra.run.smithi178.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-05-15T13:23:51.396 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-05-15T13:23:51.396 INFO:teuthology.orchestra.run.smithi178.stderr:real 0m7.296s 2024-05-15T13:23:51.396 INFO:teuthology.orchestra.run.smithi178.stderr:user 0m27.208s 2024-05-15T13:23:51.396 INFO:teuthology.orchestra.run.smithi178.stderr:sys 0m0.943s 2024-05-15T13:23:51.439 INFO:teuthology.orchestra.run.smithi117.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-05-15T13:23:51.954 INFO:teuthology.orchestra.run.smithi117.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-05-15T13:23:51.955 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-05-15T13:23:51.956 INFO:teuthology.orchestra.run.smithi117.stderr:real 0m7.884s 2024-05-15T13:23:51.956 INFO:teuthology.orchestra.run.smithi117.stderr:user 0m37.017s 2024-05-15T13:23:51.956 INFO:teuthology.orchestra.run.smithi117.stderr:sys 0m1.322s 2024-05-15T13:23:51.956 INFO:tasks.ceph:Archiving logs... 2024-05-15T13:23:51.956 DEBUG:teuthology.misc:Transferring archived files from smithi117:/var/log/ceph to /home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707111/remote/smithi117/log 2024-05-15T13:23:51.957 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-15T13:23:57.077 DEBUG:teuthology.misc:Transferring archived files from smithi178:/var/log/ceph to /home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707111/remote/smithi178/log 2024-05-15T13:23:57.078 DEBUG:teuthology.orchestra.run.smithi178:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-15T13:24:01.143 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_ae2481253f9b2fa3451954e615eedb67ebdd5314/qa/tasks/ceph.py", line 1935, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_ae2481253f9b2fa3451954e615eedb67ebdd5314/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_ae2481253f9b2fa3451954e615eedb67ebdd5314/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-05-15T13:24:01.145 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-05-15T13:24:01.153 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_ae2481253f9b2fa3451954e615eedb67ebdd5314/qa/tasks/ceph.py", line 1935, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_ae2481253f9b2fa3451954e615eedb67ebdd5314/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_ae2481253f9b2fa3451954e615eedb67ebdd5314/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-05-15T13:24:01.154 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-05-15T13:24:01.154 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-15T13:24:01.157 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-15T13:24:01.248 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-05-15T13:24:01.257 INFO:teuthology.task.clock:Checking final clock skew... 2024-05-15T13:24:01.257 DEBUG:teuthology.orchestra.run.smithi117:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-15T13:24:01.260 DEBUG:teuthology.orchestra.run.smithi178:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-15T13:24:01.275 INFO:teuthology.orchestra.run.smithi117.stderr:bash: line 1: ntpq: command not found 2024-05-15T13:24:01.277 INFO:teuthology.orchestra.run.smithi178.stderr:bash: line 1: ntpq: command not found 2024-05-15T13:24:01.280 INFO:teuthology.orchestra.run.smithi117.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-15T13:24:01.280 INFO:teuthology.orchestra.run.smithi117.stdout:=============================================================================== 2024-05-15T13:24:01.280 INFO:teuthology.orchestra.run.smithi117.stdout:^* hv01.front.sepia.ceph.com 3 6 377 61 +1182us[+1173us] +/- 48ms 2024-05-15T13:24:01.280 INFO:teuthology.orchestra.run.smithi117.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 65 -1128us[-1136us] +/- 67ms 2024-05-15T13:24:01.280 INFO:teuthology.orchestra.run.smithi117.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 62 -756us[ -765us] +/- 89ms 2024-05-15T13:24:01.280 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T13:24:01.282 INFO:teuthology.orchestra.run.smithi178.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-15T13:24:01.282 INFO:teuthology.orchestra.run.smithi178.stdout:=============================================================================== 2024-05-15T13:24:01.282 INFO:teuthology.orchestra.run.smithi178.stdout:^* hv01.front.sepia.ceph.com 3 6 377 62 +1183us[+1164us] +/- 48ms 2024-05-15T13:24:01.282 INFO:teuthology.orchestra.run.smithi178.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 1 -1119us[-1119us] +/- 68ms 2024-05-15T13:24:01.282 INFO:teuthology.orchestra.run.smithi178.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 2 -763us[ -763us] +/- 90ms 2024-05-15T13:24:01.282 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T13:24:01.283 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-05-15T13:24:01.292 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-05-15T13:24:01.293 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-05-15T13:24:01.301 DEBUG:teuthology.orchestra.run.smithi117:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-15T13:24:01.390 DEBUG:teuthology.orchestra.run.smithi178:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-15T13:24:01.457 DEBUG:teuthology.orchestra.run.smithi117:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-05-15T13:24:01.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T13:24:01.487 DEBUG:teuthology.orchestra.run.smithi178:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-05-15T13:24:01.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T13:24:01.517 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-05-15T13:24:01.525 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-05-15T13:24:01.534 INFO:teuthology.task.internal:Duration was 3438.846568 seconds 2024-05-15T13:24:01.534 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-05-15T13:24:01.543 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-05-15T13:24:01.543 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-15T13:24:01.546 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-15T13:24:01.584 INFO:teuthology.orchestra.run.smithi117.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-15T13:24:01.598 INFO:teuthology.orchestra.run.smithi178.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-15T13:24:01.952 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-05-15T13:24:01.952 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi117.front.sepia.ceph.com 2024-05-15T13:24:01.952 DEBUG:teuthology.orchestra.run.smithi117:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-15T13:24:01.973 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi178.front.sepia.ceph.com 2024-05-15T13:24:01.973 DEBUG:teuthology.orchestra.run.smithi178:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-15T13:24:02.021 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-05-15T13:24:02.021 DEBUG:teuthology.orchestra.run.smithi117:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-15T13:24:02.024 DEBUG:teuthology.orchestra.run.smithi178:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-15T13:24:02.090 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-05-15T13:24:02.091 DEBUG:teuthology.orchestra.run.smithi117:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-15T13:24:02.093 DEBUG:teuthology.orchestra.run.smithi178:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-15T13:24:02.234 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-05-15T13:24:02.243 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-05-15T13:24:02.243 DEBUG:teuthology.orchestra.run.smithi117:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-15T13:24:02.268 DEBUG:teuthology.orchestra.run.smithi178:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-15T13:24:02.294 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-05-15T13:24:02.304 DEBUG:teuthology.orchestra.run.smithi117:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-15T13:24:02.311 DEBUG:teuthology.orchestra.run.smithi178:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-15T13:24:02.332 INFO:teuthology.orchestra.run.smithi117.stdout:kernel.core_pattern = core 2024-05-15T13:24:02.357 INFO:teuthology.orchestra.run.smithi178.stdout:kernel.core_pattern = core 2024-05-15T13:24:02.384 DEBUG:teuthology.orchestra.run.smithi117:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-15T13:24:02.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T13:24:02.400 DEBUG:teuthology.orchestra.run.smithi178:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-15T13:24:02.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T13:24:02.439 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-05-15T13:24:02.448 INFO:teuthology.task.internal:Transferring archived files... 2024-05-15T13:24:02.449 DEBUG:teuthology.misc:Transferring archived files from smithi117:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707111/remote/smithi117 2024-05-15T13:24:02.449 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-15T13:24:02.501 DEBUG:teuthology.misc:Transferring archived files from smithi178:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707111/remote/smithi178 2024-05-15T13:24:02.501 DEBUG:teuthology.orchestra.run.smithi178:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-15T13:24:02.534 INFO:teuthology.task.internal:Removing archive directory... 2024-05-15T13:24:02.534 DEBUG:teuthology.orchestra.run.smithi117:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-15T13:24:02.536 DEBUG:teuthology.orchestra.run.smithi178:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-15T13:24:02.585 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-05-15T13:24:02.597 INFO:teuthology.task.internal:Not uploading archives. 2024-05-15T13:24:02.597 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-05-15T13:24:02.606 INFO:teuthology.task.internal:Tidying up after the test... 2024-05-15T13:24:02.606 DEBUG:teuthology.orchestra.run.smithi117:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-15T13:24:02.608 DEBUG:teuthology.orchestra.run.smithi178:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-15T13:24:02.621 INFO:teuthology.orchestra.run.smithi117.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 15 13:24 /home/ubuntu/cephtest 2024-05-15T13:24:02.640 INFO:teuthology.orchestra.run.smithi178.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 15 13:24 /home/ubuntu/cephtest 2024-05-15T13:24:02.641 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-05-15T13:24:02.651 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-05-15T13:24:02.660 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-05-15T13:24:02.661 INFO:teuthology.nuke:Checking targets against current locks 2024-05-15T13:24:02.688 DEBUG:teuthology.nuke:shortname: smithi117 2024-05-15T13:24:02.688 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-15T13:24:02.713 DEBUG:teuthology.nuke:shortname: smithi178 2024-05-15T13:24:02.713 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-15T13:24:02.716 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi117.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707111', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-15 12:17:45.404670', 'locked_by': 'scheduled_rzarzynski@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIsXkaK4XQPzB8+ZJdXcA3a5GBSnAV9wzg7BG9USRcj7bw0CjWp7FrSJylm7crDcKiGM5FFlopish4Bjir3K9qc='} 2024-05-15T13:24:02.740 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi178.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707111', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-15 12:17:45.403394', 'locked_by': 'scheduled_rzarzynski@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPHTLb3nCf+LZPtkT+i7iba/GITUHDkKt0Jf053EyLYISLrDwZ2shMTtn0NUH8ax5kcE9uVIvY6w5Ahszapyv4w='} 2024-05-15T13:24:02.775 INFO:teuthology.orchestra.console.smithi117:Power off 2024-05-15T13:24:02.775 DEBUG:teuthology.orchestra.console.smithi117:pexpect command: ipmitool -H smithi117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-05-15T13:24:02.798 DEBUG:teuthology.orchestra.console.smithi117:power off output: Chassis Power Control: Down/Off 2024-05-15T13:24:02.798 DEBUG:teuthology.orchestra.console.smithi117:pexpect command: ipmitool -H smithi117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-15T13:24:02.813 INFO:teuthology.orchestra.console.smithi178:Power off 2024-05-15T13:24:02.813 DEBUG:teuthology.orchestra.console.smithi178:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-05-15T13:24:02.822 DEBUG:teuthology.orchestra.console.smithi117:check power output: Chassis Power is on 2024-05-15T13:24:02.839 DEBUG:teuthology.orchestra.console.smithi178:power off output: Chassis Power Control: Down/Off 2024-05-15T13:24:02.839 DEBUG:teuthology.orchestra.console.smithi178:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-15T13:24:02.863 DEBUG:teuthology.orchestra.console.smithi178:check power output: Chassis Power is on 2024-05-15T13:24:06.823 DEBUG:teuthology.orchestra.console.smithi117:pexpect command: ipmitool -H smithi117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-15T13:24:06.864 DEBUG:teuthology.orchestra.console.smithi178:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-15T13:24:06.932 DEBUG:teuthology.orchestra.console.smithi117:check power output: Chassis Power is on 2024-05-15T13:24:06.974 DEBUG:teuthology.orchestra.console.smithi178:check power output: Chassis Power is on 2024-05-15T13:24:10.938 DEBUG:teuthology.orchestra.console.smithi117:pexpect command: ipmitool -H smithi117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-15T13:24:10.975 DEBUG:teuthology.orchestra.console.smithi178:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-15T13:24:11.047 DEBUG:teuthology.orchestra.console.smithi117:check power output: Chassis Power is off 2024-05-15T13:24:11.084 DEBUG:teuthology.orchestra.console.smithi178:check power output: Chassis Power is off 2024-05-15T13:24:11.149 INFO:teuthology.orchestra.console.smithi117:Power off completed 2024-05-15T13:24:11.185 INFO:teuthology.orchestra.console.smithi178:Power off completed 2024-05-15T13:24:11.290 INFO:teuthology.run:Summary data: description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados tasks/rados_api_tests validater/valgrind} duration: 3438.8465678691864 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_rzarzynski@teuthology status: fail success: false 2024-05-15T13:24:11.290 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-15T13:24:11.389 INFO:teuthology.run:FAIL